What is ISBD?

ISBD stands for Iridium Short Burst Data. It is a system used by iridium.com to transfer data to and especially from its Iridium Satellite Constellation. This communication satellite system is able to send and receive data messages to isolated ground units which can be arbitrarily located anywhere on the surface of the earth.

How does it work?

Devices using the Iridium system can send data from remote locations (M.O. - Mobile Originate). These messages are moved through the satellite communications network in space and then to a ground station which is able to attempt delivery to the final application user. One way it can do this is to simply email the message. While this may be fine for isolated high-priority problem reports it is not robust for large sensor arrays. Eventually at scale, email will be indistinguishable from spam to intermediate networks. The Iridium relay service has a more efficient option which is to make direct TCP/IP socket connections to specially prepared servers waiting to receive such messages. The example program shown below is exactly this kind of server. The point of this module is to make this kind of server extremely simple to create.

Who would use this? Let’s say you threw some drifting sensors (like this) off the back of a ship in the middle of the ocean. The sensors measure ocean and air temperature, water salinity, maybe some other interesting properties. The unit has a transmitter that can send this data to the Iridium network. How does the data get from the Iridium network to the researchers who deployed the unit? The researchers can use this module to write custom parsers for Iridium SBD messages received by email or they can use it to create a custom server, as shown in the example below, which can receive and process the messages directly.

What about unpacking the payload data?

What makes these ISBD messages a bit tricky is that they’re packed very efficiently. The whole point of the ISBD system is to deliver to the end user some application specific data. This application data itself is also, usually, packed very efficiently per the end user’s specifications. Of course this module can’t anticipate what end users are going to require, but it does serve as a reasonable template for making a separate custom module that can decode compact application specific binary data messages into useful application data.

Copy this module, rename it to reflect your application, and change the unpacking scheme. Adjust all the other things that are specific to Iridium SBD, but generally, the structure and types of methods required will be quite similar.

Taming ISBD With Python

Why is Python a good way to handle this?

  • Excellent native network socket handling tools.

  • Excellent native threading tools.

  • Good at complicated objects with many properties and method requirements.

  • Good at working with databases and file systems.

  • Writing your own custom scripts is relatively easy and the resulting code is especially comprehensible and maintainable.

What kinds of tools can be created with this module?

  • A complete message receiving server. See isbdd below.

  • Message validators to check for errors.

  • Filters to find specific data in archived messages.

  • Utilities to build SQL databases from archived messages.

  • Client simulations to test servers using archived or synthesized messages.

Dependencies

The isbd module requires no exotic dependencies. It depends on only standard Python modules which should be universally available.

  • os

  • datetime

  • struct

  • binascii

  • socket *

  • MySQLdb *

*Optional

License

The module’s license is LGPL V.3.

Download

Here is the complete module: isbd.py (20kB)

It’s short. You can just read through it.

Usage

Simply copy the module (it’s a single file, isbd.py) in the directory with the program that will use it (or do exotic things with PYTHONPATH as you like) and import isbd. With this successfully imported into your Python program, you have access to all of the features of the Isbdmsg class.

Isbdmsg attributes

The Isbdmsg class represents the data of the received message. Class methods allow the original data to be unpacked, queried, checked, output, archived, and anything else that may relate to the ISBD message handling. Class attributes allow access to any component of the message in isolation.

Isbdmsg.entire_isbd_msg

Entire binary data blob as received.

Isbdmsg.msg_protocol_ver

Should always equal 1.

Isbdmsg.total_msg_len

Number of bytes sent by ISBD.

Isbdmsg.mo_header_iei

Should always equal 1.

Isbdmsg.mo_header_len

Should always equal 28.

Isbdmsg.cdr_ref

Call Detail Record Reference. An automatic ID number.

Isbdmsg.imei

IMEI Unit Identification.

Isbdmsg.status

Session status - 0=success; 1&2 also mostly ok.; 10,12-15=problem

Isbdmsg.momsn

Mobile Originated Message Sequence Number.

Isbdmsg.mtmsn

Mobile Terminated Message Sequence Number. Should equal 0.

Isbdmsg.msg_timestamp

Time Iridium sees msg (not arrival or unit generated). Seconds since epoch.

Isbdmsg.payload_header_iei

Should always equal 3.

Isbdmsg.payload_header_len

Should always be 11.

Isbdmsg.loc_orient

Location orientation code (0=N,E; 1=N,W; 2=S,E; 3=S,W).

Isbdmsg.loc_lat_deg

Latitude - degree part.

Isbdmsg.loc_lat_min

Latitude - minute part.

Isbdmsg.loc_lon_deg

Longitude - degree part.

Isbdmsg.loc_lon_min

Longitude - minute part.

Isbdmsg.cep_radius

Circular Error Probable (CEP) Radius.

Isbdmsg.payload_iei

Start of Payload IEI type. Should always equal 2.

Isbdmsg.payload_len

Length of this payload

Isbdmsg.payload

The actual message sent from the unit, bit for bit.

Isbdmsg.payload_hex

Printable hex string of payload.

Isbdmsg methods

The following functions can be called from Isbdmsg objects.

  • Isbdmsg.load(data) - A separate function so that users can create empty objects and load them (or reload them) with different data by calling only this function. With an argument of None this should also clear the data from a full object.

  • Isbdmsg.unpack() - Extract all possible ISBD message header data fields from the binary ISBD message blob received. If the data blob is not present (in self.entire_isbd_msg), then all of the fields will get a value of None.

  • Isbdmsg.pack() - In theory, this should be similar to unpack using the struct.pack() function. In practice, each field will have to be checked to make sure it is ready and many fields will have to be converted to the proper type. This would be useful for synthesizing new binary messages suitable for testing as well as for editing existing ones. (Not yet implemented.)

  • Isbdmsg.__repr__() - Normal Python representation string. Allows objects to be printed in a reasonably sensible way.

  • Isbdmsg.html() - HTMLized string representation of message object. Useful for showing the data on web pages and indicative of the kind of custom output features possible.

  • Isbdmsg.parts_for_output() - Returns a list of (label,value) tuples of suitable output components. Applicable to various output modes.

  • Isbdmsg.errors_check_msg() - Checks issues with the entire message itself. Returns errors found. Or None, i.e. no errors, if good.

  • Isbdmsg.errors_check_parts() - Some things should reliably be constant values in all messages. It may not be catastrophic if not, but it might be worth a log entry. Returns errors found. Or None, i.e. no errors, if good.

  • Isbdmsg.log_entry() - Return a string suitable for putting in a log file for a server. The server should add the timestamp.

  • Isbdmsg.execute_mysql(con,sql) - Connect to a MySQL/MariaDB database using the supplied connection parameters and execute the SQL. con should be a dictionary similar to this: 

    {'host':'sql.xed.ch', 'user':'xedtester', 'passwd':pw, 'db':'isbd_msg'}

  • Isbdmsg.insert_in_mysql(con) - Do an SQL "INSERT" to add this ISBD message as a record.

  • Isbdmsg.insert_in_pgsql(con) - Connect to a PostgreSQL database using the supplied connection parameters and "INSERT" this message as a record. (Not yet implemented.)

  • Isbdmsg.read_sbd_file(filename) - Load data a message blob from a file system file and unpack so that the object is ready to use with the file’s contents. This could be useful to load archived received message files for replay testing or later analysis.

  • Isbdmsg.write_sbd_file(filename) - Write this binary message blob to a file. This could be useful for writing messages received by a socket server or some other kind of acquisition mechanism or it could be used to create message files of synthesized data.

  • Isbdmsg.dated_filename(basedir,bonus="") - This will create a filename sensible for storing received messages. It will require a top level base directory, something like "/home/isbd/data/received". It will figure out a date-based subdirectory from the message metadata, not actual arrival time, allowing same day messages to be grouped properly. "/home/isbd/data/received/20161106". Note this is just a name; the directory is checked for existence on write_sbd_file(). The file name will start with the IMEI and full date. An optional bonus string (not integer) can be supplied to further identify the file; this is ideal for sending '-%06d'%mno where mno is an integer of the server’s message number. Example call and returned value. 

    M.dated_filename('/home/isbd/newmessages','-%06d'%369)
/home/isbd/newmessages/20161105/20161105180202-300234063250980-000371.sbd

  • Isbdmsg.timestamp_fmt(style="log") - Useful for using the time stamp found in self.msg_timestamp in a more practical way. The format of msg_timestamp is in seconds since the Unix epoch. This function returns a string that is more human readable. Style options are log, justdate, iso8601, mysql.

  • Isbdmsg.location_fmt(style="human") - Return a formatted string of the message location. Useful for using geographic coordinates in different applications. Styles can cover all different types of fashionable lat/long formats. See ISO-6709 for example. This function returns a string that is more human readable and/or more useful with other software. Style options are svg, lat, lon, iso6709, google, log, human.

  • Isbdmsg.send_as_socket_client(con) - Useful for testing servers. A testing program could be written that goes through a set of archived files and simulates the Iridium network using them as sample messages. Or a program could be written that creates synthetic test messages and sends them at programmed intervals. Needs connection information taking the form of a tuple of host and port. 

    con= ('isbdserver.example.edu',10800)

Typical module use

Let’s examine the parts of the server example that specifically use the module. Start by creating an Isbdmsg object. In this single line, the object is filled with data and then unpacked into its parts.

M= isbd.Isbdmsg(data).unpack()

The next line uses a utility function to generate a sensible name for a file that will contain this particular message and then the message is written to that file.

M.write_sbd_file( M.dated_filename(OUTDIR,'-%06d'%mno) )

In addition to writing the message to a file, it’s parts are also inserted into a MySQL database. Other database’s engines are possible instead or additionally.

M.insert_in_mysql(my_DB_connection_info)

Finally compose a log entry for this particular message and send it to the logging function which you can write however you like (adding time stamps, etc).

log( 'RECV:%s/%d,%s'%(ip[0],ip[1],M.log_entry()) )

A Complete Iridium SBD Message Server

The most obvious use of the isbd module is to create a server that can receive messages sent by the Iridium network’s ground station relay clients. This server is a perfect example of how to use the module and how effective it can be. To demonstrate how simple this can be I will list the entire source code, less than 75 lines. The server, called isbdd for ISBD Daemon, is complete and ready to receive messages. It is multi-threaded and can handle thousands of connections a minute, perhaps a lot more.

isbdd.py
#!/usr/bin/python
# isbdd - Iridium Short Burst Data Server
# Chris X Edwards <isbd@xed.ch>  - 2016-11-06

# Usage:
#     isbdd | tee ${SOMELOGFILE}
# What PID is listening?
#     lsof -i :10800

import isbd
import socket
from thread import *
import datetime

OUTDIR= '/home/isbd/data/received/' # Obviously set this to make sense.
PORT= 10800  # 10800 is the normal port, but can be changed for dev testing.
HOST= ''     # Server interface to bind to. Blank is `INADDR_ANY`.
BACKLOG= 20  # Max connections on accept queue. See notes.
LOGQ= list() # Need a global queue so fast log posts don't garble each other.

def log(m):
    '''Log uses real "now" (arrival) time, not ISBD message or payload time.'''
    ts= datetime.datetime.now().strftime('%Y%m%dT%H%M%SZ') # ISO8601
    LOGQ.append( ts+':'+m ) # Append is a thread-safe operation.

# == Connection Handling For Each Thread ==
def servicethread(connection,mno,ip):
    READ_BYTES= 2048 # Optimize per application.
    data= ''
    while True:
        rdata= connection.recv(READ_BYTES)
        data+= rdata # This may not be necessary, but allows for huge messages.
        #print 'Connection said: %s' % data
        if not rdata:
            break
        M= isbd.Isbdmsg(data).unpack()
        M.write_sbd_file( M.dated_filename(OUTDIR,'-%06d'%mno) )
        M.insert_in_mysql({'host':'sql.xed.ch','user':'xedtester','passwd':'xxxxxxxxx','db':'isbd_msg'})
        #M.insert_in_pgsql(con):
        log( 'RECV:%s/%d,%s'%(ip[0],ip[1],M.log_entry()) )
    connection.close()

# == Create Socket ==
s= socket.socket(socket.AF_INET, socket.SOCK_STREAM) # I.E. (IPv4,UDP)
log( 'Socket Creation OK' )
# == Binding ==
try:
    # This line is to prevent "Bind failed! Address already in use (error #98)"
    # Which occurs if you restart the server too quickly.
    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    s.bind((HOST,PORT))
except socket.error as msg:
    log( 'ERROR: Bind failed! %s (error #%s)' % (msg[1],str(msg[0])) )
    s.close()
    raise SystemExit
log('Socket Binding OK')
# == Listening ==
s.listen(BACKLOG)
log('Socket Listening on %d OK' % PORT)
# == Client Transaction ==
msg_num= int(0)
while True:
    while len(LOGQ): # Clear out queue log, i.e. don't write log concurrently.
        print LOGQ.pop(0)
    try:
        msg_num= (msg_num+1)%1e6 # Reset to 0 after a million.
        conn,addr= s.accept()
        #log( 'Connected to %s:%d' % addr )
        start_new_thread(servicethread, (conn,msg_num,addr) )
    except KeyboardInterrupt:
        break
s.close()
log('Socket Closed OK')
Notes about Python socket programming

SBD Message Details

Table 1. Schedule of Parts Of Unpacked ISBD message
Byte Function Byte Within Segment

0

version (should always = 1)

1

total message length

2

MO Header IEI (should always = 1)

3

MO DirectIP head length (should always = 28)

4

CDR Reference (an automatic ID number)

(1,2,3,4)

5

IMEI First byte of a 15 byte sequence.

(5)

…

19

15th and last byte of the IMEI sequence.

(19)

20

Session status - 0=success, 1&2 also mostly ok. 10,12-15= problem

(20)

21

MOMSN - Message Originate Message Sequence Number

(21,22)

22

MTMSN - Not message terminated so… (should always = 0)

(23,24)

23

Time of session in epoch time (4 bytes)

(25,26,27,28)

24

Start of Payload IEI type - MO Location Information IEI (should always = 3)

25

Length of this payload in 2 byte unsigned short (should always = 11)

26

Location orientation code (0=N,E; 1=N,W; 2=S,E; 3=S,W)

(1)

27

Latitude - degree part

(2)

28

Lat. minute part (2 bytes)

(3,4)

29

Longitude - degree part

(5)

30

Lon. minutes part (2 bytes)

(6,7)

31

CEP Radius (4 bytes)

(8,9,10,11)

32

Start of Payload IEI type - MO payload IEI (actual) (should always = 2)

33

Length of this payload (2 bytes)

34

DATA First byte

…

N

DATA Last byte, where N is 33 plus the value of part 33 (i.e. payload length)

An Example Message
$ xxd my_msg.sbd
00000000: 0100 4501 001c 8317 5a9e 3330 3032 3339  ..E.....Z.300234
00000010: 3939 3939 3939 3939 3900 2d85 0000 56cf  062959960.-...V.
00000020: aacc 0300 0b01 20c2 6375 3265 0000 0000  ...... .cu2e....
00000030: 0200 1500 0000 0000 0000 0000 0000 0000  ................
00000040: 0000 0000 0000 0000                      ........
Table 2. Full Explanation of Example’s Decoding
Byte Value Parts Unpack Code Explanation

1

01

c

0

Obligatory "1" - version

2

00

H

1

2nd+3rd = unsigned short value

3

45

.

1

In this case it looks like x45 = 69d (total message length)

4

01

c

2

MO Header Information Element Identifier (IEI)

5

00

H

3

with next byte is unsigned short for length

6

1c

.

3

0x1c is 28 dec, always 28 more bytes in the MO DirectIP header

7

83

I

4

CDR Reference (Auto ID number) 0-4294967295

8

17

.

4

9

5a

.

4

10

9e

.

4

10000011 00010111 01011010 10011110 = 2 199 345 822

11

33

c

5

"3" Start 15 bytes of IMEI char data

12

30

c

6

"0"

13

30

c

7

"0"

14

32

c

8

"2"

15

33

c

9

"3"

16

39

c

10

"9"

17

39

c

11

"9"

18

39

c

12

"9"

19

39

c

13

"9"

20

39

c

14

"9"

21

39

c

15

"9"

22

39

c

16

"9"

23

39

c

17

"9"

24

39

c

18

"9"

25

39

c

19

"9" -→ This example is "300239999999999".

26

00

c

20

Session status - 0=success, 1&2 also mostly ok. 10,12-15= problem

27

2d

H

21

MOMSN - Message Sequence Number 00101101 = 45 dec

28

85

.

21

Total=11653 dec 10000101 = 133 dec

29

00

H

22

MTMSN - This is not an MT, so no MT Message Sequence Number

30

00

.

22

MTMSN - This is not an MT, so no MT Message Sequence Number

31

56

I

23

01010110 Time of session 4 bytes of unsigned int epoch time

32

cf

.

23

11001111 Decimal total = 1456450252

33

aa

.

23

10101010 $(date --date="1456450252") =

34

cc

.

23

11001100 "Thu Feb 25 17:30:52 PST 2016" Correct!

35

03

c

24

Start of Payload IEI type - MO Location Information IEI is code 3

36

00

H

25

Length of this payload 2 byte unsigned short

37

0b

.

25

0x0b = 11 bytes of this location data to follow.

38

01

c

26

Start 7 byte Location (0=N,E; 1=N,W; 2=S,E; 3=S,W)

39

20

c

27

Latitude in degrees (0x00=0, 0x5a=90) This is 32 degrees

40

c2

H

28

Lat. Minutes MSB \ Range is 0 - 59999 0x0000 - 0xea5f

41

63

.

28

Lat. Minutes LSB / Where 59999 is 59.999 min. 0xc263= 49.763

42

75

c

29

Longitude in degrees (0x00=0, 0xb4=180) This is 117 degrees

43

32

H

30

Lon. Min. MSB \ Range 0 - 59.999 -→ N32d49.763', W117d12.901'

44

65

.

30

Lon. Min. LSB / 0x3265= 12.901 -→ Clairemont Mesa & Regents Rd

45

00

I

31

Start 4 byte unsigned int of CEP radius

46

00

.

31

radius in km around the "center point"

47

00

.

31

I think this is just a way to say 8 of 10 times it’ll be

48

00

.

31

no farther from the distance specified. Not used apparently.

49

02

c

32

Start of Payload IEI type - MO Payload IEI (actual) is code 2

50

00

H

33

Length of this payload 2 byte unsigned short

51

15

.

33

0x15 = 21 bytes of this payload data to follow.

52

00

c

34

data 0-

53

00

c

35

data 8-

54

00

c

36

data 16-

55

00

c

37

data 24-

56

00

c

38

data 32-

57

00

c

39

data 40-

58

00

c

40

data 48-

59

00

c

41

data 56-

60

00

c

42

data 64-

61

00

c

43

data 72-

62

00

c

44

data 80-

63

00

c

45

data 88-

64

00

c

46

data 96-

65

00

c

47

data 104-

66

00

c

48

data 112-

67

00

c

49

data 120-

68

00

c

50

data 128-

69

00

c

51

data 136-

70

00

c

52

data 144-

71

00

c

53

data 152-

72

00

c

54

data 160-
Unpack Codes

  • > = Byte order (big endian I think)

  • c = char 1 byte 0-255 (256 values)

  • H = unsigned short 2 byte 0-65535 (65,536 values)

  • I = unsigned int 4 bytes 0-4294967295 (4,294,967,296 values)

  • . = In the table above this shows another byte being used for the previous field.

These need to match:

packformat= '>cHcHIccccccccccccccccHHIcHccHcHIcH' + 'c'*(l-51)
l=list(packformat); print l.count('H')*2 + l.count('I')*4 + l.count('c')
print len(self.entire_isbd_msg)

Acronyms

CEP

Circular Error Probable

DR

Call Detail Record

CRC

Cyclical Redundancy Check

CDR

Call Detail Record

DB

Database

DSC

Delivery Short Code

DSD

Data Set Download (aka DSDR)

DSDR

Data Set Download Response (aka DR)

DSS

Diagnostic System Services

DTE

Data Terminal Equipment

ECS

ETC Communications Sub-system

ETC

Earth Terminal Controller (ETC consists of ECS, ETS & ESS)

ETS

ETC Transmission Subsystem

FA

Field Application

GBS

Gateway Billing Subsystem

GEO

Geographical (as used in ‘geographical location’)

GIE

Gateway Infrastructure Equipment

GSM

Global System for Mobile Communication

GSS

Gateway SBD Subsystem

GW

Gateway

IE

Information Element

IEI

Information Element Identifier

IMEI

International Mobile Equipment Identifier

IP

Internet Protocol

ISU

Iridium Subscriber Unit (NAL Research’s modems and trackers)

LBT

L-Band Transceiver

MO

Mobile Originated - I think this is us, sensors to base

MOM

Mobile Originated Message

MOMSN

Mobile Originated Message Sequence Number

MT

Mobile Terminated

MTM

Mobile Terminated Message

MTMSN

Mobile Terminated Message Sequence Number

SBD

Short Burst Data

SEP

Short Burst Data ETC Processor

SPNet

Iridium Service Provider Network Provisioning Tool

SPP

Short Burst Data Post Processor

VA

Vendor Application

Alternatives

Don’t like my way of doing things? As I discover them, I’ll list other projects that provide similar functionality.

Known Errors

I found this error which needs to be solved. I suspect unpacking returning none causes this.

Unhandled exception in thread started by <function servicethread at 0x7ffcd30be9b0>
Traceback (most recent call last):
  File "./isbdd", line 55, in servicethread
    M= isbd.Isbdmsg(data).unpack()
  File "/export/home/isbd/isbd_server/isbd.py", line 23, in __init__
    self.load(data)
  File "/export/home/isbd/isbd_server/isbd.py", line 31, in load
    self.unpack()
  File "/export/home/isbd/isbd_server/isbd.py", line 43, in unpack
    m= list(struct.unpack(packformat,self.entire_isbd_msg)) # m is message component list.
struct.error: unpack requires a string argument of length 51

Copyright © 2016 - Chris X Edwards