Json file for data analysis
응답
17. 9. 25 오후 3:31

Hello there.


I am trying to find a software or program to get json file of sensors in the XDK 110 for data analysis.

 

I have tried the Bosch XDK Portal and it shows the graphs etc, but not the json file.

 

Other than that I have also tried the Bluemix Watson IOT as followed in this link https://www.embarcados.com.br/conectando-bosch-xdk-sensor-ibm-bluemix/ and I did get the 3 json files as shown in the website, but I am wanting to get the json files as together in a file providing all the sensors information, and then use it for data analysis. 

 

I also think that AWSEducate can provide me with what I need, but then once again, I have the account but I do not know how to start. If there is a guide on it I would be more than happy.

 

Basically, I am trying to get the sensors data of the Bosch XDK 110 in json format, as a file and use it for data analysis. 

 

Hope someone here can help me. Thanks in advance. 


Danial 

0 (0 투표)
RE: Json file for data analysis
응답
17. 9. 29 오전 10:54 as a reply to Danial Rohmat.

Hello Danial,

The Bosch XDK Portal demo creates a string containing all sensor values in the JSON format and publishes it to "BCDS/XDK/single/%s/out/stream", where "%s" is replaced by your XDK's WLAN MAC address (it is formatted as "FF:FF:FF:FF:FF:FF"). If you want to get the raw, unmodified data, you would need to set up your own MQTT client, subscribe to the topic the Bosch XDK Portal demo used to publish to the Bosch XDK Portal broker and then you would receive all sensor data in JSON format.

There are MQTT libraries for many of the common programming languages, such as Java or Python. Additionally, a JSON library should be used, to parse the JSON easily.

As far as I know, the IBM Bluemix application from Felipe Neves does the same, but it used the MQTT Paho demo, instead of the Bosch XDK Portal demo. But the original MQTT Paho does not create a JSON object.

Regarding your question about how to receive the sensor data of the XDK as JSON file. You could build your own application, which initializes and stores the data of the sensors of the XDK. Additionally, you could extract the necessary implementation part, which contains the JSON creation from the Bosch XDK Portal demo inside the file bxpSensor.c and insert it in your application. Afterwards, you would only need to choose one transfer protocol, such as HTTP, MQTT, LWM2M, UDP or any other of the transfer protocols supported by the XDK, and send your sensor data JSON file to your client.

Whichever protocol you use, you will have to use your own effort to combine the features that the XDK offers. There is no such thing as a function that implements your exact use case. For any information regarding the protocols, the sensors and the XDK in general, you may refer to the Guides in the learning section here, and if there are specific questions to XDK-related topics (protocol implementations, FreeRTOS, etc), we are always glad to help.

Please let me know if that was helpful and do not hesitate to ask if you have further questions.

Kind regards,
Franjo

0 (0 투표)
RE: Json file for data analysis
응답
17. 9. 29 오전 3:29 as a reply to Franjo Stjepandic.

Hello there. 

 

By setting up my own MQTT client, does this mean I can make use of the MQTT Paho and subscribe to the topic Bosch XDK Portal demo and receive all the sensor data in JSON format?

 

Best regards

Danial

 

0 (0 투표)
RE: Json file for data analysis
응답
17. 9. 29 오후 4:27 as a reply to Danial Rohmat.

Hello Danial,

You can, of course, make use of the MQTT Paho libraries I linked to set up your own MQTT client and subscribe to the Bosch XDK Portal broker. As such, it should deliver all sensor data in JSON format for further proceeding.

Please tell me if that was helpful and feel free to ask if you have further questions.

Kind regards,
Franjo

0 (0 투표)
RE: Json file for data analysis
응답
17. 10. 6 오전 7:33 as a reply to Franjo Stjepandic.

Hi. 

 

Sorry for the late reply. How about AWSEducate? I have an account and I would then like to make use of it to analyze the sensors data in my XDK 110 using it. Is there a way to do this as I do not know how to start too. Thanks.

 

Danial

0 (0 투표)
RE: Json file for data analysis
응답
17. 10. 6 오후 1:19 as a reply to Danial Rohmat.

Hello Danial,

as far as I know, the AWS offers an MQTT broker, as such, you can send the data from the Bosch XDK Portal demo to the AWS MQTT Broker.

Please tell me if this was helpful, and do not hesitate to ask further questions.

Kind regards,
Franjo

0 (0 투표)
RE: Json file for data analysis
응답
17. 10. 8 오후 2:43 as a reply to Franjo Stjepandic.

Hello there.

 

Thanks for the prompt answer. You have been helping me a lot and I've been progressing at a stable rate too. Really thankful for that.

Anyways, for the AWS part, I have created a MySQL DB instance and it is already running. How do I make the XDK 110 to show it's sensors data on it then for data analysis? Which example should I flash into the XDK to make this work? I really have no clue on this as I have 0 experience on AWS as of this moment. Please advice and thank you so much!


Danial

0 (0 투표)
RE: Json file for data analysis
응답
17. 10. 9 오후 3:32 as a reply to Danial Rohmat.

Hello Danial,

MySQL DB is a database. It does not run any logic, except for handling requests. The XDK does not currently have API to directly connect to a database, but you can still make it work by adding a layer between database and XDK.

I recommend that you set up a Virtual Machine on AWS as this additionally layer. AWS offers a tutorial for that here

On that Virtual Machine, you can run an MQTT broker, using the command line tool mosquitto. This can be installed via the command $ sudo apt-get install mosquitto, once you’re connected to the virtual machine.

The program mosquitto is essentially a broker which you can start from the command line interface. You can find detailed documentation here, regarding the configuration and usage.

Your XDK can connect to the broker on your virtual machine and send data to a certain topic, for example xdk/sensors/send. You can also use the old Bosch Portal Demo application. You simply have to set your virtual machine’s address, instead of the bosch broker address.

To actually access and write data to your MySQL DB, you also need a client on the virtual machine that subscribes to the broker, receives data on that topic and inserts it into the DB accordingly.

For receiving MQTT data (subscribing to the broker), you can use the MQTT Paho Python module. For connecting and manipulating the database, you can use the MySQL Connector/Python module (or any other, for that matter). Additionally, since you would want to work with JSON, the json library is worth taking a look at.

If Python is not your preferred programming langue, you can use any other. Paho has MQTT implementations for most common languages and MySQL and JSON libraries are generally available for most languages, too.

Please let me know if this works for you and do not hesitate to ask if anything is indefinite.

Kind regards,
Franjo

0 (0 투표)
RE: Json file for data analysis
응답
17. 10. 9 오후 5:36 as a reply to Franjo Stjepandic.

Hi

I had just finished setting up the Virtual Machine on AWS as offered in the tutorial you've attached, successfully. Thanks for that. But, I do not understand how to run an MQTT broker using the command line tool mosquitto. I've searched online for solutions but to no avail too. Please advice and thanks.

Danial

0 (0 투표)
RE: Json file for data analysis
응답
17. 10. 10 오후 1:06 as a reply to Danial Rohmat.

Hello Danial,

I assume that you used an Amazon Linux AMI as your configuration. As such, when you connect to the Virtual Machine via your command terminal, you can use the following commands to install mosquitto and to launch it (a $ indicates that it is one line/command in the terminal):

$ sudo wget http://download.opensuse.org/repositories/home:/oojah:/mqtt/CentOS_CentOS-7/home:oojah:mqtt.repo -O /etc/yum.repos.d/mqtt.repo
$ sudo yum install mosquitto mosquitto-clients
$ mosquitto &
$ mosquitto_sub -t "your/topic"


The first command will update the packages you can install (this will include mosquitto, so you can install it)
The second command will install mosquitto (the broker) and client modules (for subscribing and publishing via command line)
The thirds command will start the broker
The fourth command will subscribe to the topic your/topic. You do not have to use this line, as it only serves debugging purposes (i.e. you see incoming messages on that topic in your console.

By default mosquitto and mosquitto_sub use the localhost as the host-address and the port 1883 (standard mqtt-port). In contrast, the only port open by default is 22 for the ssh connection to your VM. You will have to enable the port 1883, which you can do following the instructions here. I recommend opening the TCP port 1883 for every IP (source should be 0.0.0.0/0), otherwise you need to find out your XDK's public IP address.

You can now publish messages to the broker by using your VM's public IP address as the broker address in the Bosch Portal Demo. You can determine your VM's IP as seen in this AWS documentation section.

As a reminder, the Bosch XDK Portal demo publishes data to BCDS/XDK/single/%s/out/stream, where "%s" is replaced by your XDK's WLAN MAC address (it is formatted as FF:FF:FF:FF:FF:FF and is listed on the bottom of your XDK).

You can also find another use case for mosquitto here.

Please tell me if this was helpful and do not hesitate to ask further questions.

Kind regards,
Franjo

0 (0 투표)
RE: Json file for data analysis
응답
17. 10. 10 오후 2:42 as a reply to Franjo Stjepandic.

Hi there.

Thanks for the reply. I was following your steps to install mosquitto, but once I reached to step 3, the ($ mosquitto &), there was an error and I have attached it in the screenshot below. 

Is it supposed to be:

$ sudo su instead of $ sudo & ?

And after that what should I do when the # appears after the $ disappears? 

Please advice and thanks!

Danial

0 (0 투표)
RE: Json file for data analysis
응답
17. 10. 11 오후 2:58 as a reply to Danial Rohmat.

Hello Danial,

Your screen indicates that you did not install mosquitto yet. You only installed mosquitto-clients (for subscribing and publishing). Please run the following command to install mosquitto (the broker):

$ sudo yum install mosquitto

As additional information: sudo allows a user to execute commands with root privileges. Root privileges are, for example, needed to install applications.

The & makes it possible for the current command to execute and run, while the terminal can still be used for new commands. The command $ mosquitto would block the terminal, while $ mosquitto & allows to execute further commands.

As an alternative, you can of course use two terminals. One only running as a MQTT broker, and the other subscribing to the topic for debugging purposes

In any case, you're nearly there! Did you already open the port 1883 for your XDK?

Kind regards,
Franjo

0 (0 투표)
RE: Json file for data analysis
응답
17. 10. 16 오후 2:44 as a reply to Franjo Stjepandic.

Hi there 

Thanks for the answer. I have followed the steps until mosquitto &, where the "error" in the attached picture took place. What does this mean? I am really sorry about this I am quite blur as I have 0 experience previously. Thanks in advance.


Danial

0 (0 투표)
RE: Json file for data analysis
응답
17. 10. 17 오후 4:26 as a reply to Danial Rohmat.

Hello Danial,

The error indicates, that the mosquitto broker is already running. To stop it, you can use the command:

$ sudo service mosquitto stop

Afterwards, you can restart it again with $mosquitto&.

Please do not hesitate to ask if you have further questions.

Kind regards,
Franjo

0 (0 투표)
RE: Json file for data analysis
응답
17. 10. 17 오후 6:08 as a reply to Franjo Stjepandic.

Hello.

 

As stated by you before:

By default mosquitto and mosquitto_sub use the localhost as the host-address and the port 1883 (standard mqtt-port). In contrast, the only port open by default is 22 for the ssh connection to your VM. You will have to enable the port 1883, which you can do following the instructions here. I recommend opening the TCP port 1883 for every IP (source should be 0.0.0.0/0), otherwise you need to find out your XDK's public IP address.

I have now completed the step as you said above, but just to reconfirm before proceeding, is the attached screenshots the ones I am supposed to do, and what the outcome is supposed to be?

 

After, you said this too:

You can now publish messages to the broker by using your VM's public IP address as the broker address in the Bosch Portal Demo. You can determine your VM's IP as seen in this AWS documentation section.

The problem now is, I do not know where to get the broker address in the Bosch Portal Demo. Is it the one as I have attached below too?

Hope these helps you in guiding me. 

 

Thanks

 

Danial
 

0 (0 투표)
RE: Json file for data analysis
응답
17. 10. 18 오후 2:18 as a reply to Danial Rohmat.

Hello Danial,

Your setting for inbound rules is nearly correct, but you need to change the port for the rule. 22 is the port for SSH. I recommend that you select Custom TCP Protocol from the dropdown menu in the column Type, and then set Port Range to 1883. This is the common MQTT port (MQTT communicates using TCP).

You can retrieve the VM's public IP address on the same page. In the menu on the left-hand side select Instances. You'll see a list of your virtual machines, which probably has only one row. The column IPv4 Public IP contains the IP address.

Use this IP to replace the string "mqtt.bosch-si.com" in the code you showed in your third picture.

Kind regards,
Franjo

0 (0 투표)
RE: Json file for data analysis
응답
17. 10. 18 오후 4:22 as a reply to Franjo Stjepandic.

Hello.

I've done what u told me to do, and all seems to go well. I have attached a screenshot on what I changed, and also I already changed the IP from mqtt.bosch.si. Loaded it on the XDK and flashed it. But how do I see the sensors data? I have no idea on this at all. 

Thanks in advance!

Danial

0 (0 투표)
RE: Json file for data analysis
응답
17. 10. 19 오후 2:43 as a reply to Danial Rohmat.

Hello Danial,

First of all, you should switch the ports of the two rules. I.e. the rule with source 0.0.0.0/0 should have the port 1883 and the rule with your public IP as the source should have 22 (which is the port for SSH, the tool you connect to your VM with). At the moment you have the settings that everyone can connect on port 22, but only your PC can connect on port 1883. You need it the other way around.

Regarding how you can see your sensor data, first of all, once your XDK connects to the mosquitto broker, there should be a log in the terminal you connected to your VM with, which has mosquitto running. The log should say something like this:

1508402078: New connection from xdk-ip-address on port 1883.
1508402078: New client connected from xdk-ip-address as client-name (c1, k60).

If this log appears, you can open another terminal, with which you connect to the VM. Once you are connected, you can run the following command in the new terminal, to subscribe to the XDK's output:

$ mosquitto_sub -t "BCDS/XDK/single/XX:XX:XX:XX:XX:XX/out/stream"

Once you successfully subscribed and see the sensor data of your XDK, you can move on to your initial goal of saving that data in the MySQL Database.

Please tell me if this was helpful, and do not hesitate to ask further questions.

Kind regards,
Franjo

0 (0 투표)
RE: Json file for data analysis
응답
17. 10. 19 오후 5:13 as a reply to Franjo Stjepandic.

Hi there.

Thanks for the help! It really works and now it extracts .json data out of the XDK and I am able to read it. But now, how am I supposed to get it for data analysis? As in to read the real temperature, etc which is updated regularly.

I also have a website that I had already created which I want to add in this data analysis in and if you have a way to do that please enlighten and advice me on it. Thanks! 

Danial

0 (0 투표)
RE: Json file for data analysis
응답
17. 10. 20 오후 2:34 as a reply to Danial Rohmat.

Hello Danial,

You can program your own client in a language of your choice now, to connect to the broker on your VM. You can use any language you want. It basically depends on the specific purpose. If you want to develop a website, you are bound to languages that build on HTML. But to be honest with you, you will need to write code.

I know that you are not very experienced in that regard, but most of this can be done with only basic knowledge of programming, and a lot of googling (i.e. looking up examples on the internet). I would recommend to look into basic programming tutorials about HTML, javascript and maybe frameworks like bootstrap to quickly visualize your data.

Kind regards,
Franjo

0 (0 투표)