Sending high rate data
Answer
9/20/19 10:33 PM

Hello,

In my application I have the sensor configured to store high rate data at certain instances and store them locally in SD card (since we cannot send high rate data directly over cloud)

Now I want to send this high rate data collected in csv file to a raspberry pi for further processing.

Can anyone guide me on what wifi protocol should I look?

Or how to approach this problem

0 (0 Votes)
RE: Sending high rate data
Answer
9/23/19 6:12 AM as a reply to Hiren Patel.

Hello,

May i know at what data rate are you sampling sensor data ?

0 (0 Votes)
RE: Sending high rate data
Answer
9/23/19 6:31 AM as a reply to Padmapriya G.

Hi,

Presently I am collecting at every 3 ms (about 333.33hz).

But that is only at particular instances for about 1.5 mins. And that data is stored in SD card.

Rest of time it is running at low speed rate and not storing data in SD card.

 

So once a particular time the high speed data is collected and stored, I want it to be sent to raspberry simulating cloud.

0 (0 Votes)
RE: Sending high rate data
Answer
9/25/19 7:58 AM as a reply to Hiren Patel.

Hello,

You can use protocols like HTTP/ MQTT. There are template applications in XDK workbench, which you can refer to send data to server.

https://developer.bosch.com/web/xdk/application-templates

Please choose protocol according to your server run in raspberry . For example, you can sent sensor data as HTTP post message to the HTTP server running in the raspberry .

 

0 (0 Votes)
RE: Sending high rate data
Answer
9/29/19 7:50 PM as a reply to Padmapriya G.

Hello,

Thanks for the reply,

I am planning to use HTTP POST request to send the data.

can anyone tell me the maximum size of payload that can be sent from xdk.

I know XDk has limited memory so we cannot send a big  payload.

0 (0 Votes)
RE: Sending high rate data
Answer
9/30/19 11:50 AM as a reply to Hiren Patel.

Hello Hiren,

according to your application (streaming of sensor data) i would highly reocmmend the MQTT protocoll instead of HTTP. MQTT is much more fitting for this kind of use case.

Anyways, regarding to your question... if you are using the REST HTTP client you have a payload limit of about 50 byte. If you are using the HTTP client there is no such a limit. But i think buffer larger then 500 bytes are not suitable for the XDK environment.

If you have any further question don't hesitate to ask.

Best regards,

Christian

 

0 (0 Votes)
RE: Sending high rate data
Answer
9/30/19 12:19 PM as a reply to Christian Köhler.

Hi,

Thank you for the advice.

I plan to use rasberry pi as mqtt broker and send data from xdk to pi.

Problem is I still would be needed to first store the data in SD card and then send to rasberry as uploading data simultaneously would slow the data rate right?

So m question is how do i send the data from xdk (like in text format, JSON) and I would need to send the data in parts by succesively reading the .txt file right?

0 (0 Votes)
RE: Sending high rate data
Answer
9/30/19 2:07 PM as a reply to Hiren Patel.

Hello Hiren,

In my experience, the opposite is exactly the case.  Publishing data over MQTT allows a much higher performance then writing them to the SD card.

In my application i used the XDK to capture data from an external Sensor via SPI bus. My sampling rate was 10 kHz. I captured the data and stored them in 500 byte sized buffers and published these buffers in MQTT messages. The system worked pretty fine for me.

In my experience the bottle neck to performance was always the SD card. So if you can avoid the use of the SD card in your application there should be no performance problems with your mentioned data rates.

I hope this was helpfull.

Best regards,

Christian

0 (0 Votes)
RE: Sending high rate data
Answer
9/30/19 3:15 PM as a reply to Christian Köhler.

Hello,

Thank you so much for the feedback, i really underestimated the power of mqtt.

May i have a look of your code. I want to get the idea of handling buffer and bulk update.

Also if you used rasberry or other broker then code for that might also help

thank you

0 (0 Votes)
RE: Sending high rate data
Answer
10/1/19 7:48 AM as a reply to Hiren Patel.

Hello Hiren,

first my application from back then: I used the XDK as MQTT client and a computer as MQTT broker. The broker was implemented with Visual Studio.

Then the buffer algorithm. Mostly I used the buffer algorithm from the example program "DemoDataLogger" with some modifications (see pseudocode below).

The MQTT stuff i looked in the example project "SendDataOverMqtt". In my application i used the API Serval_Mqtt.h directly because then you have more flexibility.

In the pseudocode below you can see the rough structure. In reality for sure you have to do some additional stuff. Especially FreeRTOS operations. You have to run two different tasks. One of the is collecting your sensor data, the other one publishes to collected data over MQTT. Both tasks need to be synchronised via Semaphores.

I hope this was helpfull. If you have any further questens don't hesitate to ask.

Best regards,

Christian

 

typedef struct {

char data[512];
uint16_t length;
}

// Declare Buffers
Buffer pingBuffer;
Buffer pongBuffer;
Buffer *activeBuffer;
Buffer *backBuffer;

void InitBuffer(void){

activeBuffer = &pingBuffer;
backBuffer = &pongBuffer;
}

void ResetBuffer(void){

activeBuffer->length = 0;
activeBuffer->data[0] = 0;
backBuffer->length = 0;
backBuffer->data[0] = 0;
}

void SwitchBuffer(void){

backBuffer->length = 0;
backBuffer->data[0] = 0;

if(activeBuffer == &pingBuffer){

activeBuffer = &pongBuffer;
backBuffer = &pingBuffer;
} else {

activeBuffer = &pingBuffer;
backBuffer = &pongBuffer;
}

activeBuffer->length = 0;
activeBuffer->data[0] = 0;
}
}

void SomeCallbackWithYourData(uint16_t data){

activeBuffer->length += sprintf(activeBuffer->data + activeBuffer->length, "%u", (unsigned short int) data);

if(activeBuffer->length >= 490){

SwitchBuffer();
PublishMqtt();
}
}

void PublishMqtt(void){

MQttPublish("yourtopic", "yourqos", backbuffer->data);
}

void main(void){

InitBuffer();
ResetBuffer();
}

 

0 (0 Votes)