Elektor Lab Notes 26: Data Loggers, IoT Storage, Data Recovery, and More
on
Welcome back to Elektor Lab Notes! Every few weeks, our engineers and busy editors at Elektor post a few lab notes and updates about new DIY electronics projects, tech industry news, and helpful engineering tips. In this edition of Lab Notes, we talk about data loggers, recovery of data from corrupt SD cards, and more. Please share your thoughts in the Discussion section at the bottom of this page. Feel free to post your own lab notes and let us know what you are working on at your electronics workbench!
Jean-François Simon (Engineer, Elektor)
Data Recording and Long-Term Logging: A few years ago, I suddenly felt the urge to measure and record the electricity consumption of my home over long periods. At the time, my colleague Saad hadn’t yet created his ESP32 Energy Monitor project, but many solutions for this purpose already existed, such as the powerful Open Energy Monitor based on the Raspberry Pi. However, the desire to reinvent the wheel being always strong, I wanted to build my own solution. The energy meter installed by my electricity provider had a serial data output, so all I had to do was add a system to format the data (voltage, current, total consumption in kWh) and store it. I used an OpenLog module from SparkFun, along with a small circuit to convert the meter’s output signal (a 50kHz amplitude-modulated sine wave) to the 3.3V logic format compatible with the logger. The rest was the power supply section, reusing the PCB of a simple 1-cell 18650 USB power bank, to continue logging (or shut down properly) in case of a power outage.
After several years of loyal service, the micro-SD card I was using began to show signs of failure. Although the card was still recognized by Windows, trying to open a log file would make Notepad and Windows Explorer crash with a “Not Responding” message. Trying to copy the file to another hard drive produced the dreaded “I/O error”, otherwise known as a photographer’s nightmare. Yet neither the command-line (chkdsk) nor the graphical version of Windows’ built-in disk checking tool found any errors.
“No further action is required.” Well… I don’t think so! On the contrary, the time had come to move to more advanced methods. I had used iCare SD Card Recovery in the past for situations like this. Unfortunately, this time the little free program was of no help. Strangely, it didn’t even recognize the SD card’s partition. There are many software tools that can recover photos from SD cards, but I didn’t use them because most are paid and optimized specifically for recovering accidentally deleted photos, reconstructing the headers of JPG files. PhotoRec by CGSecurity can also recover text files, but it’s not the easiest to use since it also retrieves deleted files or those that existed before formatting. In this case, I chose to use ddrescue.
After creating a Linux Mint bootable USB stick, ddrescue can be installed using the command: sudo apt install gddrescue. Insert the faulty SD card into the reader; Linux Mint will mount it automatically. Be sure to unmount it first. Open a Terminal and run the following command:
sudo ddrescue -f -n -d /dev/sdb sd.img sd.log
The -f option tells ddrescue to overwrite the output image file if it already exists. The -n option instructs it not to retry reading bad sectors, for a faster first pass. The -d option enables direct access to the device, bypassing the kernel's cache. The source disk, in this case the SD card, is /dev/sdb. The recovered data is written to an image file named sd.img. Make sure you have enough space, as the image file will be the size of the full SD card (16GB in this case). I used a Windows (NTFS) partition as the destination, to easily retrieve the file later.
Let ddrescue run as long as needed, and make sure your computer doesn’t go to sleep. After a full night and 5 passes, ddrescue was able to recover 99.97% of the data. Then, all that remains is to mount the sd.img file. On Windows, there are various methods for this, as an example it can be done very easily using ImDisk. A virtual drive appears, containing a copy of the SD card’s contents after recovery with ddrescue. This makes it easy to access the recovered files. As the file I was particularly interested in is rather large (more than 800 MB), I recommend using a dedicated tool such as Large Text File Viewer to open it smoothly.
Now one question remains: what to do with all this data? I haven’t found a ready-made answer. To be able to browse the data, zoom in and out, and take measurements, a tool like Grafana might be suitable. But the data must first be imported from text format into an appropriate database. It’s a work in progress, so stay tuned for future updates. These days, Wi-Fi and online storage are all the rage, so local storage is no longer as commonly used for this type of data logger as it once was. Flash memory, FRAM, SSDs, USB drives... If you’ve built a logger in the past, what solution did you use? Let us know in the comments.
C. J. Abate (Director, Content and Engineering)
Over the past few weeks, the Content and Engineering team has been collaborating with the Elektor Academy Pro team on developing specialized learning products for electronics professionals and engineering teams. Elektor Academy Pro team's mission is to create comprehensive hands-on resources that address real-world challenges in electronics and embedded systems. The engineers and editors on our team are supporting the expansion of Elektor Academy's professional offerings, including Pro Boxes, expert-led webinars, and industry conferences. In the coming months, we will be contributing technical content for Elektor's B2B publications aimed at helping professionals stay current with technological advancements.
In other news, we've been diving deep into artificial intelligence. Due to our going fascination with AI, our team at Elektor is currently exploring several exciting initiatives, with a particular focus on integrating AI technologies into our production workflows. We're investigating various AI tools that can enhance our content creation process, streamline workflow management, and improve our capability to scout for breaking industry news and trends. This exploration aims to keep Elektor at the cutting edge of technical publishing while maintaining our commitment to high-quality educational content for electronics enthusiasts.
AI solutions:
- The team is evaluating AI-powered content production tools to optimize article development and technical documentation
- We are discussing the possibility of implementing new AI-based workflow management systems to improve team coordination and project tracking. We are looking into developing an automated news-scouting framework to identify emerging trends in electronics and technology.
- We are testing AI tools specifically designed for electronics documentation and circuit design validation.
If you have ideas or recommendations, feel free to share them with us!
Discussion (0 comments)