I am Kevin Hubbard of Black Mesa Labs. I am a High Altitude Space Balloon Engineer and this is my story of the last 3 days:
I work with a small group of ultrasound engineers in Issaquah,WA-USA who are also the Balloongineers – 1 of 7 teams in Washington State that participate in the Global Space Balloon Challenge. There are 398 teams across the planet that participate each year in the challenge.
Last month I was invited to join the local Issaquah team. Someone from my past must have told the Balloongineers that I worked on the fault tolerant AS/400 at IBM in the early 90’s ( descendant of the Space Shuttle’s AP-101 Flight Computer ). I certainly wasn’t recruited into the Balloongineers for any high altitude bravery. As a quick introduction to the team as the greenhorn they asked me to use all of my Black Mesa Labs’ Maker resources to design and build a new electronics module for Flight Navigation, Data recording and live 2-way Satellite Communications to earth – all in 6 weeks. Go Big or Go Home – sounds fun!
If you would prefer to skip the story, there are YouTube videos at the end of the story. Technical details for HAB1 including block diagram and descriptions are at the very end of this blog. Grab a cup of coffee.
— Story Begins —
I spent 6 challenging weekends non stop doing Hardware and Software development and Test to make the launch date. HAB1 electronics are built from a high altitude flight ready uBlox GPS unit, a 2-way Iridium RockBLOCK satellite communication modem, 1/2 a dozen custom circuits boards design and built using CopperConnection and OSH-Park, a fully custom FPGA “Lizard Brain” design and a RaspberryPi Linux computer running a custom Flight and Communication program written in Python. All these pieces had to work together – all the time – with built in fault tolerance – all in 6 weeks for the contest opening.
Sunday : Final hardware assembly 04.24.2016. The 3 10Ah batteries are 132g or about 5 oz each and make up about 1/2 of the weight of the electronics package. The entire electronics package including the batteries weighs in at 870g ( 31oz or almost 2 lbs ). This does not include the styrofoam sphere, 3 solar panels and 3 LEDs. Not bad for a 32bit Linux computer, 30Ah of batteries, GPS, 2-Way SatCom and a custom FPGA board.
Rain tests with the RockBLOCK 2-way Iridium SatCom modem in my driveway did NOT look promising. Driving tests on Sunday even worse. Monday took a long route into work with my convertible with the top down and only 1 in 4 messages got through driving around at 35 MPH ( 60 km/h ) – but the launch window for the contest was this week. I am confident SatCom will work much better when high above the clouds. RockBLOCK can fail or work intermittently in flight I told myself – only the uBlox GPS unit has to work perfectly to feed altitude information to the Flight Computer for fully autonomous operations of flight control for helium venting and ballast pumping. So long as the Flight Computer has reliable flight data, downlink telemetry to Earth can be flaky. OneBusAway has to work reliably all the time – balloon telemetry? balloons are slow – who cares if we miss a few? This isn’t a rocket after all – so I thought.
The Balloongineers Team Leader Jon picks a launch date based on weather predictions for Wednesday – We Go! Final HAB1 pieces all brought together per block diagram.
Wednesday : Launch Day 04.27.2016 – Launch site – south end of Lake Sammamish,WA. A large crowd formed on the grassy lawn while the low pressure gas specialists begin filling the HAB1 balloon from a 50L tank containing helium at 2,200 psi or 7,500L ( 2200psi * 50L / 14.7psi = 7483 liters of helium ). The helium begins the slow transfer and expansion to partially inflate the 2000g Kaymont balloon to 2m diameter at 15 psi. I connect power to the Lizard Brain and then boot the Flight Computer. It takes about 10 minutes to boot the Flight Computer ( Linux bootup followed by GPS lock time and then finally POST ) – so I need to get all of my systems running and checked out in advance of launch. Once GPS is locked, the Flight Computer finishes Power On Self Test by exercising the helium vent valve for 3 seconds and then the ballast pump for 2 seconds. Payload estimate for the 3 vessels was 9 lbs, which required 6123 liters of helium based on remaining 400 psi in the tank ( 400psi * 50L / 14.7psi = 1360 liters remaining ).
Ten minutes later as expected, POST has finished with the comical site of 2 seconds of ballast liquid squirting out a straw onto the ground. Flight Computer is now a Go for launch.
Low pressure gas expert and valve designer Dr.P and I are now both waiting for the now fully inflated balloon to attach to the valve assembly ( which is connected to the Flight Computer capsule via a 3 wire servo connection ). As the 4″ hose clamp is tightened by the fuel specialists and balloon handlers – we gasp in horror ( caught on video verbally and by the look of terror on my face here ) as we realize the small 3 pin 0.100″ pitch SIP connector connecting the Flight Computer to the Valve servo assembly is not only disconnected – but the pins are bent. SIP connectors are great – I use them all the time – but they really don’t like to be bent. These were dangerously close to the hose clamp. Bend SIPs once – maybe, bend them twice – and they break off. Do we abort the launch? Is there time to solder a new SIP connector on the leads running 12″ to the flight computer? No – the balloon is filled and ready to go. The balloon handlers in their special balloon protecting gloves are now responsible for keeping the balloon untouched from any foreign objects before flight – it just is not feasible to abort this late in launch with the balloon already inflated.
We must fix what we have. I grab a pair of pliers ( plumber’s pliers mind you – not precision needle nose 22 gauge wire pliers the Flight Computer was built with ) that are on the table for hose clamp assembly. I carefully bend the pins back. Dr.P in charge of the Valve Assembly inspects my work – I plug it into the Valve assembly. The connector is not polarized ( it can be connected backwards – and definitely not work ) and the hose clamp is covering the pin labeling. Is the polarity correct ? Yes. Will it hold? Duct tape it – it must hold. Can we retest the valve preflight? No time, that would either take a full power cycle of the Flight Computer to re-run POST ( 1o minutes ) – or a sprint to ground control ( my laptop ) to type a SatCom message for manual venting. Manual SatCom commands take anywhere from 15-30 minutes to schedule and be applied depending on the current SatCom window. Iridium packets aren’t fast and aren’t inexpensive. Neither retest method are viable. Visual inspection only it has to be – it looks good. Okay – Flight Computer and Valve Assembly are both Go for launch. Balloongineers we are a go for Launch!
The Balloongineers launched at 2:30PM PST, I mean 21:30 Zulu – I am still the team greenhorn. HAB1 launches with my flight computer capsule, a video+SPOT capsule and finally the ballast vessel.
Our backup SPOT GPS ground recovery system failed 5 minutes into flight and immediately we became completely dependent for recovery on the primary Iridium 2-way RockBLOCK SatCom modem and my Flight Computer software and custom hardware design. This was all my new custom design – it ALL had to work now – pressure was on – but I was confident in my work and really was not that concerned. Maybe we should use APRS as the backup system next time – whatever. Primary system is functioning perfectly now and it will work on the ground for recovery. Live detailed telemetry is coming in at anticipated 10-15 minute intervals. Voltage, Current, Temperature, Ascent Rate all Green across the board.
Autonomous picture taken at 109 meters ( 360 feet ) above Sammamish State Park
Autonomous picture taken at 455 meters ( 1,500 feet ) above the Issaquah Highlands
Autonomous picture taken at 1120 meters ( 3,700 feet ) going north up Lake Sammamish
One hour into flight SatCom is still 5×5, reporting in every 10-15 minutes with detailed live telemetry data as scheduled. We were ascending at about 500 ft/min as planned and reached 60K feet per FAA requirements to not hang out in commercial air space for too long. Ascent rate a little high, but manageable. Major ascent correction was not scheduled until 75K feet anyways. Traveling North – ever so slowly.
Autonomous picture taken at 9,772 meters (32K feet ). Note that the Flight Computer inserts the top line telemetry overlay text to each JPEG image while in flight. The CPU looks up nearest city from live GPS coordinates using an on board database of all cities in North America. One of many benefits of using Linux+Python. Numbers on the right are system temperatures, voltages and currents. -3C outside, +25C inside at the batteries. Solar panels generating 228mA at 4.5V. Systems all look great. Insulation is working.
Around 00:00 Zulu a regular telemetry downlink message came in – at 80K feet now, but ascent rate of 600 ft/min. Too fast. Something isn’t right, Flight Computer should have autonomously adjusted by venting excess helium in 5 second bursts to slow ascent from 500 ft/min to only 250 ft/min by now. It should have reported this venting back to ground control – it didn’t. Critical scheduled GPS location messages aren’t all getting through though. At 30K feet there was a 25 minute gap between RockBLOCK messages. By (my) design the low priority “! V” venting messages – indicating short 5 second helium vents are posted writes ( fire and forget ) – and are not retried like recovery critical GPS location messages. It is conceivable that the lack of any vent messages is just due to the unreliable SatCom issues. All of a sudden – these low level diagnostic “! V” messages became extremely important. Other than at poweron POST – we haven’t seen any. Was the electrical SIP connector connection to the valve broken pre-launch? Was the Flight Computer malfunctioning? Altitude is still okay though. I am not fully alarmed yet. Waiting for next anticipated message around 00:10 to 00:15 Zulu to determine if venting is working or not. The next message doesn’t come. No “! V” – no GPS location. Silence. Regularly scheduled SatCom 10 minute telemetry update came and went. Dudes – something isn’t right ( this isn’t NASA after all ) – come here quick. HAB1 has not reported in.
Autonomous picture taken at 35,832 meters (85K feet )
Core group of Balloongineers gathers around ground control ( my laptop ). We quickly do some math. Last report of 80K feet and +600 ft/min means 100K ( anticipated balloon burst altitude ) in just 30 minutes from when the last message was received. Mission plan was to hover around 90K feet and hope for some favorable winds to take us east ( there weren’t any ). Uplink command reception and processing from Earth to Space are slow ( 15-30 minutes ) as the HAB has to check for them, and its SatCom window is 10-15 minute when all systems are working flawlessly. Now they obviously were not. Our window to act manually is closing and closing fast. I am starting to question my SIP connection repair to the valve. Maybe the pin broke in flight with the extreme temperature drop to -50C. Core group of us take a vote and all agree – attempt manual override as we have no indication the autonomous flight computer has vented. How much? Maximum. I quickly type “open 45” on my keyboard and double check my syntax and spelling ( yes – even for open ) then click <SEND>. This queues up a manual command via the Iridium Satellite network to override the autonomous Flight Computer and vent excess helium for 45 seconds. 45 seconds – a big vent compared to planned short 5sec burst vents by the autonomous Flight Computer but we need to do this NOW! It is done – now we can only wait. There is no immediate ACK. The message will now sit in an Iridium mailbox system until the HAB1 checks in at regular SatCom intervals and asks if it has any messages for it. These regular SatCom intervals have been quite irregular however – oh and HAB1 by design only checks for uplinks every OTHER downlink message as every uplink check costs an Iridium credit whether you have a message or not. I was trying to be a good steward of Jon’s Iridium credits. I am now regretting that design decision.
Autonomous picture taken at 26,413 meters (87K feet )
We know now the manual intervention to command a helium vent for 45 seconds was in fact received by HAB1, but unfortunately came in a bit too little, too late as it turns out. Ascent rate was slowed – but not enough, and the HAB kept rising. Regularly scheduled downlink telemetry messages were not coming in. At 5:30pm The Flight Computer issued one last disturbing downlink message home while in flight to ground control on Earth saying “My batteries are fully charged, my electronics are at +20C, but I Have Just FAULTED and REBOOTED myself, I am here ( precise GPS coordinates for a 100×100 foot spot in near space 100,000 feet above Mill Creek ).”. Not good. Not good at all.
Autonomous picture taken at 30,835 meters (101K feet )
This was the last the Balloongineers heard from our little High Altitude Balloon in flight. Not a very promising message at that. All was quiet. Semi regular 10 minute telemetry downlink via Iridium was severed. Ground control was completely in the dark. Where is HAB1?
Wednesday night, I slept very little. I checked for new messages from above – but received none. I scoured through the 12 short 50 character messages received during the 3 hour flight but saw no answers to all of my questions. The “! BOOT action_satcom()” message told me a reboot had occurred and that the software was running action_satcom() at the time of the reboot. To have the Lizard Brain force a watchdog reset meant the CPU must have been stuck inside action_satcom() for almost 10 minutes. This function call should take 3 minutes at the most. How? What? Why?
2:38PM N47.5577 W122.0484 M18,+0 C+22,+27,+28,+26 ! BOOT 0 0
2:45PM N47.5543 W122.0466 M455,+0 C+20,+28,+29,+27
2:49PM N47.5497 W122.0448 M1120,+125 Issaquah WA
2:56PM N47.5485 W122.0459 M1809,+120 C+16,+27,+29,+26
2:59PM N47.5518 W122.0494 M2473,+120 Issaquah WA
3:07PM N47.5617 W122.0582 M3155,+139 V3.9,204,4.5,148
3:10PM N47.5722 W122.0753 M4136,+139 West Lake Sammamish
3:14PM N47.5808 W122.0846 M4690,+130 C+16,+28,+29,+26
3:22PM N47.6053 W122.1088 M5895,+158 C+14,+31,+33,+30
3:47PM N47.7463 W122.2280 M9772,+186 Inglewood-Finn Hill
4:03PM N47.8086 W122.2639 M13060,+163 C-4,+25,+23,+19
4:14PM N47.8092 W122.2390 M14700,+165 Kenmore WA
4:20PM N47.8045 W122.2346 M15315,+165 C-7,+23,+21,+16
4:23PM N47.8126 W122.2186 M16322,+189 Bothell WA
4:33PM N47.8100 W122.1974 M18034,+163 V3.9,263,4.5,227
4:49PM N47.8120 W122.1724 M20299,+164 V3.9,214,4.4,83
4:52PM N47.8116 W122.1660 M21378,+212 Maltby WA
4:56PM N47.8140 W122.1678 M22010,+180 V3.9,259,4.5,197
5:03PM N47.8169 W122.1698 M23323,+198 V3.9,263,4.5,229
5:08PM N47.8129 W122.1788 M24585,+186 V3.9,266,4.5,203
5:35PM N47.8225 W122.1751 M29772,+0 C+12,+22,+23,+15 ! BOOT action_satcom() 9878, 10790
Analyzing the SatCom telemetry data – all systems were green – Voltage, Current, Temperature – all better than could be expected for the harsh near space environment at the time of reboot. My worst case planned for scenario was that a sustained -50C outside environment could bring the Lithium Ion batteries down to -20C and shut down all systems. The FPGA Lizard Brain was designed to prevent this. It continuously monitors battery voltage, current and temperature and has a means to warm the batteries via power resistors in between the cells. The last report were batteries were a warm +22C and fully charged at 3.9V with a predicted current draw of 200mA. Power definitely was not it. Single Event Upset due to excessive alpha ray particles, I planned for that too – everything but GPS and RockBLOCK was wrapped in thick aluminium foil for maximum heat retention, radiation hardening and RF shielding. What happened? Why did HAB1 go silent?
Thursday : I drove in to work and told myself – the balloon must have popped above 100K, the parachute deployed ( as designed ) – it is safely on the ground somewhere around Mill Creek ( or maybe …. Puget Sound ? ). HAB1 will have recognized descent and autonomously exited flight mode and entered recovery mode based on altitude. Recovery mode has a single mission – preserve battery power and phone home – Winter is Coming. The FPGA Lizard Brain will boot the CPU once an hour and phone home with GPS coordinates and then immediately go back to sleep for an hour. This process should start with each day’s rising sun hitting the solar panels. Autonomously ( I modeled my design after the Mars Sojourner rover ). But it didn’t. I myself was very quiet on Thursday. HAB1 was well, completely silent. Not a peep – all day long. No hourly phone home message – nada. Puget Sound I’m thinking – I bet that is far worse for the RockBLOCK than a rainy drive in my car.
Friday : I drove in to work feeling down – really down. Our balloon was down and we didn’t know where other than it most likely dropped via parachute from 100K – 120K feet above Mill Creek. Big search area. If we were NASA we would simply ask the NSA to retask one of their birds in the sky. We were neither NASA nor NSA however. The balloon itself was one time use expendable, the 2-way RockBLOCK satellite modem, not exactly a one time use disposable item. I arrived at work and started writing a short email as a small gofundme like fund raiser for the purchase of a replacement Iridium RockBLOCK modem to give us one more shot at this year’s competition. The RockBLOCK is a pricey little piece of hardware and I was not okay with Jon buying a 2nd unit himself just because our 1st HAB was lost in space ( or Earth as it may be ). Sparkfun has them in stock ( I just checked ) and amortizing a replacement modem across 10-20 space enthusiastic engineers – all children during NASA’s 1960-1970s Mercury , Gemini and Apollo glory days seemed a logical, moral option. I also could not shake this ridiculous notion to send a HAB2 up to try and locate HAB1. I clicked send – and waited.
Literally minutes later I received an email message – at 8:13AM- not from a generous donor however – but from our humble little HAB1 space Balloon reporting “It is 8:13AM – I am here in Mill Creek,WA at GPS …, … at 300 feet elevation. Please come get me. It is +12C and I am cold” (okay – I embellished a bit on that last part, it did report the +12C, battery status ( still at 80% after 3 days ) and precise GPS location and time – honest ).
! V 3
! B 2
I just about fainted. I definitely yelled out ( my neighbors are used to it by now ). I called Terry and Jon the team principals and we jumped in Terry’s truck, made a quick stop at his house for mountain climbing gear( for Mill Creek,WA ? my sister lives there – there aren’t any mountains ) and lots of long poles ( who has this many long poles? ).
We headed North during rush hour to a precise location on the ground ( or so I thought ). Now was definitely not the time to complain or question the ridiculous I-405 toll lane tolls.
107 Meters – I’m picturing in my head our bird on the ground on a nice grassy 300 foot hill surely – nope – not even – after a reasonable in country boots on the ground search area ( Terry also brought 3 pairs of rubber galoshes ) we located our little balloon on top of a tree. A REALLY TALL tree. This picture shows the red capsule with the GoPro and DashCam ( center ) and about 1/2 the tree. Tree grew out of the bottom of a ravine. No snakes at the bottom – but it sure wouldn’t surprise me at this point.
So while Terry goes completely Rambo down one side of the ravine, Jon and I go door to door on the other side meeting really nice neighbors of Mill Creek. Jon knocks on doors and explains our situation ( Jon is REALLY good at this by now – 3rd year at it) and politely asks for permission to enter and walk through their back yards. Everyone is pleasant, interested and just darn nice. We ~picked~ a GREAT landing neighborhood I tell myself. We locate a decent backyard breach point into the steep ravine by ~our~ tree – the REALLY tall tree, growing from the bottom of a really steep and deep ravine. I’m thinking – oh well, at least we know where it ended up – its in a better place now and at least not lost out to sea.
Terry is all like, hey we’re here – lets have a go at it. I protest – as of 8:12am Sparkfun has 66 RockBLOCKs in stock and I am certain they don’t carry Terrys ( Terry does have a brother – but it just would NOT be the same ). I lose this argument – it turns out my considerable voting weight for electronics design and flight control software does not transfer over 1:1 in country for recovery operations votes. Not even close. Fine.
So I proceed to watch our friend Terry ( who I remind you – we have only one of ) climb 75 feet up a tree clearly climbing to his death leaving me to explain to his wife how could I? Oh – and Terry had his truck keys on him to boot. I like taking Ubers as much as the next guy – but NOT to my friend’s house to explain to his wife what he died for – $500 of misc one-off electronics stuffed inside a styrofoam capsule full of telemetry data, pictures of too many clouds and some GoPro video. Oh – and would you mind please giving me a ride back to my car when you are all done crying? I just do not need this today.
Terry clearly should have brought extra shoes ( the kind with spikes ). I really should have brought extra pants ( the kind with extra absorption ). You know those moments in time where time seems to stop? You just know something really bad is about to happen and there is absolutely nothing you can do to prevent it? Lets just say it was a really long 2 hours watching Terry climb that tree. I think I will just stay at ground control next time and get started on the Python and Verilog improvements for next launch thank you very much.
To make a long story even longer – Terry survived. He is still with us still today. His two small children still have a father and not a styrofoam space capsule in his place. Colleen can wake up to Terry’s smiling face instead of a solar panel and 1 Hz flashing Navigation strobe LEDs. Terry climbed that ridiculously tall tree ( with safety rope and a carabiner ) and managed to knock the balloon down with a really long ( 15′ ) stick he saw lying in the middle of the really tall tree. It was like watching a Pinata get whacked by a friend on the edge of a cliff over shark infested waters facing certain death and oh by the way – there is no candy to shower down at the end. The styrofoam HAB capsules came down – Jon picked them up, we thanked the nice neighbor Phil ( who took all these great pictures and graciously emailed them to me ) and we were on our way home. Oh, and we did wait for Terry to climb down the tree – he had the only truck keys after all.
— Main Story Ends —
— In Closing —
In closing I just want to say how pleased I am with how many accomplishments Team Balloongineers achieved with HAB1 04.27.2016. For the 1st time we successfully vented excess helium via the Dr.P valve and slowed an ascent ( not enough and not autonomously – but we achieved a major 1st ). For the 1st time we successfully carried liquid ballast aloft and autonomously pumped this ballast to try and slow a descent ( didn’t have a balloon at the time – but still a major 1st – the Flight Computer pumped ballast ). For the 1st time we had semi-reliable downlink telemetry data reporting GPS position, altitude and system status live in flight. Up until descent – we knew exactly in space-time where HAB1 was. For the first time we had a GPS unit capable of working at 100,000 feet and provide accurate position and altitude. For the 1st time we generated electricity in flight with great success above the cloud layer ( 1W at 60,000 feet). For the 1st time we had a HAB capable of night flight per FAA regulations. For the 1st time we demonstrated we could take an actual computer with a file system aloft under battery power and use an advanced language ( Python ) for doing advanced computing in flight real-time like GPS City Lookups, Voice Synthesis, NMEA coordinate conversions, JPEG EXIF information embedding, JPEG text overlaying of telemetry data. For the 1st time we had an advanced power management and fault tolerant “Lizard Brain” FPGA overseer of all operations keeping all systems running. And what I think is the biggest 1st – for the first time, we had remote uplink control of a high altitude balloon 100,000 feet above us in real time. Team Balloongineers typed “open 45” in Issaquah,WA on the ground and at 100,000 up in the air above Mill Creek, HAB1 dutifully obeyed our command and vented excess helium for 45 seconds. Had we instead type “open 90” maybe we would have flown for another day – who knows.
Please enjoys Jon’s fully iMac edited and dramatic GoPro video on YouTube here
Please enjoy the short 6 minute time lapse video made from 5MP RaspPi camera below. Telemetry data was annotated live in-flight by the Pi and is overlayed at the top of each image with time, location, altitude, GPS coordinates, Temperature, Voltage and Current for HAB1. Music performed by the great late David Bowie.
Please enjoy the short 10 minute DashCam video below of the 3 hour flight to 103K feet and the dramatic parachute descent to ground.
Post Flight Analysis:
All of the retrieved electronics are in perfect working order. The log_gps.txt reported a peak altitude of 31,361 meters ( 102,864 feet = 19.5 miles ) at 00:44:10 Zulu on 04.28.2016. Tons of detailed telemetry data collected of our balloon adventure to near space in the log_primary.txt.
As I suspected, Battery and FPGA were completely golden throughout the flight. Paranoia and insulation was spot on. Hardware Team done good! The flight computer failed to autonomously control ascent however. Software Development and Software Test Teams – maybe some heads should roll over this – oh wait – same head. Never mind!
Post flight calculated ground speed unfortunately indicates we stopped moving once we got above 10km (30,000 feet). At 10km (30K feet) we were doing 60kmh ( 40 mph ) and then above that slowed down to under 10kmh ( 5mph ). Spent 2/3 of the flight at walking speeds – we were going to be stuck in Mill Creek,WA for a while ( normally I would say unfortunately – but my sister lives there – so I just won’t say anything).
Notice the 13 minute gap starting at 00:20:13 Zulu. CPU has been rebooted due to watchdog timeout with action_satcom(). 7 minutes after the reboot it receives and processed the “open 45” vent command while at 102K feet. The balloon has vented for 45 seconds, but will rise another 1,000 feet in the next 3 minutes to finally burst at 103K feet . The Flight Computer had just rebooted and was unable to calculate ascent after the reboot. Looking at the altitude logs and calculating after the fact, the 45 second helium vent slowed the ascent from 172m/min to 150m/min. Should we have vented longer than 45 seconds? Probably not, autonomous vents are capped at 45 seconds for a reason. Venting longer risks exceeding the 10 minute watchdog timeout and could result in a HAB1 flight computer reboot. Reboots and risky and take time. 45 seconds was the right number. Command was just executed too late.
We did not travel east as we had hoped to. That was disappointing. There was no wind and we traveled up and hung around Mill Creek for some time before bursting at 103K feet. To be clear, the balloon was going to pop eventually. A weather balloon that starts out at 2m in diameter at launch can expand to a diameter of up to 10m at near outer space. As the balloon climbs to the edge of space it eventually expands to the point where it bursts. We were hoping to delay this burst as much as possible and hover between 70K and 90K under autonomous Flight Computer control – maybe next time. The view at 100K – spectacular.
There was no ONE major failure ( ask any rocket scientist – there rarely are ) . There were a bunch of minor failures that compounded on each other. My one-off HAB1 design includes 2,000 lines of Python for the RaspberryPi Flight Computer and 8,000 lines for Verilog for the Lattice FPGA Lizard Brain. Within these 10,000 lines of source design files I located a single extra “#” present on my part in a battery saving algorithm of the Flight Computer that was removed just days before launch. This algorithm was tested before hand so in the back of my mind I knew valve operations all worked. Unfortunately in removing the battery saving feature, the flight computer also lost the ability to call the open_valve() function. A single extra “#” pound key ( yes – one key press ) prevented autonomous ascent control from happening. Power On Self Test checks the valve on each powerup, and it still could and did call open_valve() and exercise the valve – so the hardware looked completely functional during pre-flight check. Manual SatCom initiated vents – those worked too. Sigh… extra pounds are just never a good thing in ballooning.
I design ASICs for a living, so this single character within 10,000 lines didn’t surprise me too much. Unfortunate yes – preventable yes – surprising no. Complete failure of the flight computer for autonomous navigation was specifically planned for in advance. We wanted it to work, but always expected it might not. Manual SatCom uplink commands for both venting and ballast pumping could be done – But only if SatCom was working 5×5. At 80K feet SatCom went bad. The worst of the minor failures was definitely the external RockBlock communications thread starving the 10 minute hardware watchdog timer. I never predicted this. Worst case delays for action_satcom() during all the on the ground car “flight” testing was 1 minute max, so 10 minutes for a watchdog timeout seemed the correct setting. At 500 ft/min, a 10 minute delay would result in the flight computer being DOA for 5,000 feet – barely acceptable. A watchdog timeout of 5 minutes would be dangerous given than a single action_satcom() for both uplink and downlink were observed to take 4 minutes on the ground. More than 10 minutes timeout would literally never fly. The FPGA Lizard Brain did exactly what it was designed to do and ensured the flight computer would be rebooted and brought back on-line in the event of a catastrophic software failure. This happened. The FPGA Lizard Brain Watchdog timer timed out after 10 minutes and the RaspPi was rebooted. Way to go Lizard Brain! This was a huge success, unfortunately the event occurred at the worst possible time at 80K feet during an uncontrolled ascent of 600 ft/min. This would have been a really bad time to lose flight control ( if it had been enabled ). Certainly would like to send a 2nd Pi up just for communications – but that won’t happen ( the extra pounds problem again ).
Do I regret the extra ‘#’ pound? – of course. Do I question my abilities over it? Hell no. In 6 weeks of spare time I designed and constructed a one-off fault tolerant flight control and communication system that allowed us to manually vent helium from a balloon 100,000 feet above earth – from ANYWHERE ON OR ABOVE THIS PLANET EARTH – how cool is that? Not only did it do this, but it accurately reported position, temperature, voltage and current telemetry data throughout the flight AND correctly recovered flight and communication operations after a catastrophic software failure at 80,000 feet. This desired flight plan failure has actually been a huge success. Fix what is broken, improve and repeat. Hell yeah. Did I mention telemetry data all came in on my Gmail account that I could check from my iPhone while jogging on a treadmill at the gym? Wanna bet even Wernher von Braun would be a little bit envious?
Sharing a single RaspPi serial port and time multiplexing it between the RockBLOCK and the Mesa Bus to the FPGA was a design decision made by me alone upfront to minimize payload weight and power and maximize by limited available time for software development. The ability to develop the Python Flight Control software on a desktop Win7 PC connected to the Lizard Brain FPGA over a single FTDI cable ( via Mesa Bus Protocol ) was a huge win for development schedule. In retrospect – I would design this differently of course for in-flight operations. The time sharing of a single serial port was functional and tested via hours of convertible drives on the ground -where this all worked flawlessly – but this car driving could not possibly emulate the spotty Iridium connection issues we experienced at 80,000 feet up and no matter how fast I drive my car – I can’t drive 20 miles straight up.
Next iteration HAB2 will add a mini USB Hub ( light as possible ) to the RaspPi A+ and an FTDI cable ( shortened to minimum length for minimum weight ) so that the RockBLOCK gets a dedicated serial port. Any external unpredictable delays from the externally provided RockBLOCK interface Python module will NOT be allowed to trip up the watchdog timer ( maybe kick the watchdog immediately entering and during action_satcom() instead of just once in the main control loop ? ). A reliable backup recovery system is a must of course – SPOT isn’t it – APRS being the most likely solution going forward as is completely independent of spotting birds in the sky ( APRS is terrestrial 144 MHz 2-meter RF ).
Spending the rest of this weekend examining 3 hours of flight data and 3 days of recovery data to iterate, improve and launch again. Learn from mistakes – improve the design and move forward to the next launch. Man what an adventure. Sign me up for next year ( the electronics and software that is – Terry can keep his tree climbing job ).
What Went Wrong:
- RockBLOCK did not work on the descent – AT ALL. Worst mistake made. Descent was faster than I envisioned and spinning rapidly on the vertical parachute line axis. My car testing with RockBLOCK should have been a clue that descent would be a problem. In my head I had envisioned a nice gently parachute drop – which clearly did not happen. Watching the GoPro video it is clear that HAB1 not only descended rapidly under parachute deployment but it was rotating as well. Stuck in the tree, the HAB1 electronics capsule was still spinning. The directional RockBLOCK patch antenna pointed sideways had no chance to establish an Iridium link during the fast and spinning descent. The sideways mounted RockBLOCK antenna was mandated by the inverted parachute design and lack of a gimble mechanism for all of the electronics. This was a colossal and forseeable failure. Pointing the RockBLOCK patch antenna straight up for both ascent and descent is a must for future HAB missions. A terrestrial backup recovery system such as GPS+GSM Cellular or APRS is needed. I suspect the side mounted RockBLOCK may have been partly to blame for the Watchdog reset as well.
- I disabled the ability for autonomous venting. Already tested and working software was modified last minute by commenting out a few lines of Python. These comments were never fully retested. Software Regression Testing is invaluable. One pound over.
- The RockBLOCK Python module I imported went away in a CPU no mans land for way longer than I could have ever anticipated – resulting in the watchdog timeout and CPU reboot at a critical moment in flight leaving us entirely in the blind between 80K and 100K feet. If this had not happened, we would have had sufficient time to perform multiple manual vents instead of just the single vent for 45 seconds. This must be fixed. Hopefully not sleeping the RockBLOCK and dedicating a FTDI serial connection to it will do the trick.
- SPOT backup GPS locator failed 5 minutes into flight. Backup let us down.
- RockBLOCK worked once and only once stuck up in a tree. It spent 2 days in that tree and sent only a single message out during that entire time.
- I tried to conserve Iridium credits and only check for uplink messages every other downlink. Mistake. Latency matters when traveling at 80K feet and ascending at 500 ft/min. Not the time to conserve Iridium credits. Credits are expensive and we planned for a multi-day flight. My algorithm was overly stingy.
- Inverted parachute prevented ballast pump from working during descent. The flight computer was desperately trying to pump ballast to prevent descent – but with the inverted parachute configuration, the submersible pump was now at the top of the container and unable to pump anything after a few milliliters. This surely contributed to the RockBLOCK not being able to send any messages on descent.
- I was too conservative and stingy in using Iridium credits on the ground for testing. I didn’t pay for the credits, so I was trying to be a good steward for them and conserve them for the actual flight. HUGE mistake. Should have spent Iridium credits on the ground during car ~flight~ testing like a weekend in Vegas.
- Final rescue message at 8:13am reported an FPGA 28bit ON time of 100693 seconds. This is /3,600 = 28 hours. Actual time from Wednesday launch to Friday’s message was 42 hours. This large FPGA time drift still bothers me however. The Twin Paradox could certainly explain a time delta between HAB1’s FPGA RTC and ground control if HAB1’s descent was fast enough ( it was fairly fast ) – but I am fairly certain the chute deployed prior to HAB1 approaching C unless my math is really off and I messed up a few exponents. A more plausible explanation is that the FPGA is using an internal 48 MHz RC oscillator for crude time time keeping for power control, Nav LEDs and serial interface UART baud generation. Actual flight logs all report time using rock solid Zulu time provided by GPS which is accurate to 40 ns ( crazy accurate ). This needs testing in a controlled ground environment to try and reproduce. Possibly use a real crystal oscillator next go around FPGA pin permitting.
What Went Right:
- Decision to switch from Arduino to RaspberryPi was crucial in developing this advanced SatCom capable flight computer in a short amount of time. I am easily 10x to 20x faster and efficient in writing Python versus compiling embedded C++ to a microcontroller. Early software development was done strictly on PC. Final software development was still in a way done on my PC – just via an SSH connection via Ethernet into the Linux based RaspberryPi from my desktop PC ( with the 24″ screen ). RaspberryPi was an excellent choice for this project. Only asked for 100mA – well worth it. Took nice pictures for us too.
- Decision to use a FPGA for the Lizard Brain. The Mesa Logic FPGA took care of all the misc IO that a RaspPi’s limited GPIO pins couldn’t do ( FPGA provides 24 GPIOs ). FPGA also handled real time tasks the RaspPi couldn’t handle ( Servo PWM pulse generation, Nav LED 1Hz flashing, real time clock , etc, etc ). This list goes on and on. Chip design is my thing, so designing the critical function in Verilog increased my confidence for the entire project. I am ecstatic that the CPU failed in flight and the FPGA watchdog timer recovered full flight and communications. Well – you know what I mean – I wish it hadn’t failed of course – I’m just ecstatic this fault recovery mechanism worked 100% as planned. One of my favorite parts of Apollo 13 is:NASA Director: “This could be the worst disaster NASA’s ever experienced.”Gene Kranz: “With all due respect, sir, I believe this is gonna be our finest hour.” – I feel exactly this way about the HAB-1 RockBLOCK watchdog reset. Things in software failed catastrophically, but HAB1 recovered and could phone home to be recovered.
- Decision to dump the barometer and use the U-BLOX NEO-M8N flight capable GPS instead. This unit once configured for flight mode reliably provided altitude and GPS location at 100K feet. Gem of an investment for $35.
- Decision to use Mesa Bus Protocol for the link between the CPU and FPGA. Using Mesa Bus Protocol allowed me to prototype software from a Windows PC with only a FTDI cable. Big Win for productivity. Mesa Bus Protocol emulates a 32bit PCI interface – only slower – allowing me to provide a rich 32bit register and SRAM memory space in the FPGA that any CPU could access over a simple serial interface.
- Decision to include solar charging. The 3 panels hardly weigh a thing and basically kept the 30,000 mAh batteries fully charged through the flight. Brought in 1W of solar energy above the clouds. If we manage to float through a day or two on the next flight – this will be crucial in keeping the batteries charged throughout the flight and survive a long wait for recovery. HAB1 batteries were still at 80% after 40 hours in the wild. Be prepared. Winter is Coming.
- Decision to log tons of telemetry data to a 32 GB USB flash drive.
- Insulation worked fantastic. At -40C outside during the descent, +7C was the coldest any of the electronics got ( unwrapped RockBLOCK ). The batteries wrapped in foil were still at toasty +22C.
- Decision to write the last_function() function. This Python function is called whenever a main loop sub-function is called ( such as action_satcom() ) and stores the functions name and time to a file on USB flash drive. By including this, after a reboot the CPU was able to send back to Earth in-flight that it rebooted and the reason why ( action_satcom() ). Would have been completely in the dark regarding the reboot otherwise. This little blackbox function was an awesome tool for fault analysis.
Planned Change List:
- Place the battery saving algorithm BACK in. I designed (and tested) it for a reason and I still think it was a good idea. In the event of a GPS failure to update altitude ( Very possible at 100K feet ) – the Flight Computer would have been stuck in a loop opening and closing the helium vent valve every minute. There is only one battery and this failure would have brought the entire HAB1 power supply down eventually and prevented it from ever phoning home. The algorithm was a good idea – I should have pushed back harder on agreeing to take it out. It was set for 100 minutes originally – which would have saved the battery in the event of a GPS altitude failure and still allowed complete helium venting. I was the engineer in the thick of it and I failed to push back on upper management when I should have. Lesson learned – be the squeaky wheel.
- Change the action_satcom() routine. Allow this to extend the Watchdog timer by kicking the timer inside action_satcom() multiple times. Current design kicks the timer once in the main loop ( which is the proper thing to do ). With this unknown RockBLOCK failure in the sky – need to try and prevent in from occurring again. Disabling the watchdog entirely is a bad idea – as the CPU could get stuck forever and the FPGA would never reboot it – killing the HAB.
- Stop sharing the single Pi serial port. The mux scheme appears to work – but adds risk to the design and also limits the update rate for everything for the entire HAB system. Given that a separate Flight and Communication computer is not an option ( NASA gets them ) – having a dedicated serial port for each major function is a fair compromise. I won’t say back down on this change. We add a little more weight for a USB hub and FTDI cable, draw a little more current – but get a much improved HAB system in return. Make SatCom window updates more frequent if possible. Definitely check for uplinks every downlink slot instead of every other. 05.05.2016 Footnote : Have been testing with a USB Hub and FTDI cable for a couple of days now. Current for HAB1 electronics has gone up from 200mA to 300mA with this new setup. HAB2 will increase solar power generation from 1W to 13W using these larger panels that will also form the outer shell for all the electronics. Battery capacity will be reduced (30Ah to 20Ah) to shed weight and HAB2 will be 90% Solar powered during the day and enter a duty cycle hibernation mode at night where FPGA+GPS is powered 100% and CPU+RockBlock is 20%.
- A minor thing – RaspPi still images are taken too slowly when interesting things happen ( low altitude, high descent rates ). Need to improve the photo taking algorithm to take these major events into account. Same for DashCam – although a little more complicated as FPGA controls this autonomously without commands from CPU. It could take suggestions from the CPU. A up/down servo pan mechanism would be great if light enough.
- Another minor thing – RaspPi still images are annotated in NMEA GPS coordinates. Switch to decimal degrees instead ( easier to map ) and also include ascent rates from Nav Computer. Placing City names on the far right would help keep text aligned for time lapse video post flight also.
HAB1 electronics are based on a RaspPi A+ running Raspbian Linux and a custom HAB.PY program that talks Mesa Bus Protocol over a single serial link to the Mesa Logic FPGA ( Lattice ICE5LP4K ). By using Mesa Bus Protocol, 100% early on software could be written and tested on a standard desktop PC and connected to the actual hardware using only a FTDI cable. Once the HAB.PY software was ready for “flight” testing ( in my car ), it was copied over to a RaspberryPi. Transition was just a matter of changing some Windows paths to Linux paths. The Mesa Logic FPGA interfaces to a RockBLOCK sat modem, a uBlox flight capable GPS unit, helium venting servo, ballast pumping pump and a whole bunch of temperature, voltage and current monitors and low and high side FET switches. The FPGA is the “Lizard Brain” while the RaspPi is the high level Flight and Communications Computer. The entire system is powered from a 5V main supplied by a stack of 3 10,000 mAh 3.7V nominal Lithium Ion batteries that can be recharged in flight via 3 solar panels. Everything combined is less than 3 lbs ( well not including a single extra software “#” pound ). The NMEA decoder actually decodes GPS NMEA streams and stores only valid GPS locked data into a ping-pong SRAM buffer. This allows the CPU to leisurely read the most recent GPS signal rather than having to continuously process the serial stream itself. In battery under voltage conditions, the FPGA can then still receive “last known” position information without the CPU being alive. My only complaint about the uBlox is having to write to it on powerup to configure flight mode. FPGA power management continuously watched battery voltage levels and throttles back duty cycles to preserve power. For example, then battery drops from 3.7V to 3.6V, the DashCam recorder will only record 1/2 as much as before. Eventually the FPGA shuts auxiliary devices off entirely and begins placing the CPU in a PWM mode of on for 20 minutes, off for 40 minutes. FPGAs main goal is to preserve battery power. Winter is Coming.
Mesa Logic FPGA:
Mesa Logic ( aka Mesa DIP FPGA ) is a general purpose FPGA board designed around the 4,000 Logic Cell Lattice ICE5LP4K-SG48. Mesa Logic is like an FPGA equivalent to an Arduino in that the Lattice JTAG programmer is only needed once after board assembly to load a Mesa Logic bootloader into the PROM. Once the bootloader is in the PROM, a standard FTDI cable speaking Mesa Bus Protocol and the .NET BD_SHELL.exe application is all that is needed to load new firmware bitfiles. Mesa Logic provides 24 general purpose LVCMOS 3.3V user IO on a 0.100″ x 0.600″ DIP-32 grid. For HAB1 the FPGA user firmware is the “Lizard Brain” and takes care of power management, PWM Servo control, voltage,current and temperature monitoring and serial GPS decoding of NMEA data. The FPGA decodes and stores the most recent NMEA valid ( GPS locked ) string to SRAM – allowing the CPU to sleep when idle and still have access to most recent valid GPS data. HAB1 system partitioning assigns all the low-level mission critical tasks like power management ( peripheral throttling ) and navigation LEDs to the 100% up and reliable hardware. High level tasks such as communications and navigations are assigned to the RaspPi CPU with only a 2-wire serial communication needed to connected to two talking Mesa Bus Protocol. At OSH-Park the Mesa Logic FPGA boards are located at OSH-Park under BML here. A YouTube video of the Mesa Logic FPGA board under reflow using the BML 1″ SMT Reflow Hotplate is available here:
As a BSEE, I haven’t gone by the title “Software Engineer” since 1996 when I switched from typing C full-time for embedded 68K designs to writing Verilog to infer flip-flops and combinatorial logic for ASIC and FPGA hardware designs. For the last 20 years any sequentially executed software I “code” is primarily to support my hardware designs ( ChipVault, BD_SHELL, SUMP, etc ). I will be the 1st to admit that hab.py is no perfectly engineered piece of software. I am sure there are plenty of internet trolls out there that will rip me to shreds for all the “;” I use in my Python, or my liberal use of self.* globals ( pronounced “Goebbels” as they are in fact truly evil ). All I can say is, first join an international high altitude balloon team and design part of the system – any part – in 6 weeks – then lets sit down over a beer and I will listen to everything you have to say. I include hap.py as open-source as I hope someone might find bits and pieces of it useful. I am particularly proud of the 20 lines of Python that determines live in flight the closest city in North America that HAB1 is near – all without an internet connection. It was very cool to receive email messages live from HAB1 saying “Bothell,WA” and not have to continually look up GPS coordinates. Others might find all the GPS NMEA translations and JPEG EXIF data embedding useful as well. Just don’t re-use action_satcom() as it still needs a little work. Talking to Iridium birds 500 miles above is a tricky business. The open-source eSpeak utility is incorporated into HAB.BY to voice over narrate live the DashCam video with altitude and nearest city information. The function action_cfg_ublox() is golden – as it puts the uBlox GPS unit in flight mode on bootup and configures it to reliably report GPS altitude to 150,000 feet. I certainly wrote more Python try+excepts in HAB.PY then probably all of my other Python scripts combined. Of course, the failure you don’t plan for ( a 10+ minute timeout to an external RockBLOCK routine ) is the failure that occurs in flight. In closing for this chapter on software – developing Python in a Linux environment was a joy ( stressful but still a joy ). Had my $5 PiZero contained a camera connection – I certainly would have sent it up instead. Definitely never would have made our launch date if the Flight Computer software had been written for my 32bit ARM Mesa-Duino-M0 Arduino-Zero clone as originally planned. Writing Python in a Linux environment is so much more efficient than writing embedded C.
The Custom PCBs
The ADC board uses a 74HC4051 to mux 8 analog channels ( 4 temperature, 2 voltage, 2 current ) into a single 12bit ADCS7476 ADC. The FPGA drive S[2:0] with the desired channel to convert and the SPI interface transfers the data. An onboard BU33 3.3V LDO regulates the 5V main to a 3.3V clean supply and reference voltage for the converter.
HAB1 has 4 LMT89 thermistor breakout boards for measuring outside temperate, battery temperatue, CPU temperature and SatCom RockBLOCK temperature.
A ZXCT1107 is used for high side current sensing for both the battery and solar panels. This PCB also contains voltage dividers (2:1) for measuring up to 6V from the 3.3V referenced ADC.
The Micro USB power board provides for a Micro USB power input. The board was designs to allow diode ORing multiple batteries into various rails but that was not utilized for HAB1.
The N_FET board uses 1 or 2 IRML6344 low side FETs for switching on high current devices like the ballast pump and Nav LEDs.
The high side FET PCB contains 4 NCP331 high side FETs for switching on power to CPU, Servo and uBlox GPS unit.
Mesa Logic 0.100″ (DIP ) next to Mesa Logic 0.050″. These are both multipurpose low cost FPGA boards ( ~$10 ) that I use for BML projects all the time. Programmed using a standard FTDI cable and my BD_SHELL.EXE .NET application written in PowerShell.
Very first HAB1 prototype ( yes – Breadboards do still exists and have a use).
NavLED testing. The 2W LEDs I got from Osram were blindingly bright. FPGA flashes them automatically at 1 Hz when Solar panels say it is nighttime or one a minute when landing has been declared and HAB1 enters recovery mode.
Current and Voltage sense boards. Only 2 of the 3 were used in flight. All of the HAB1 PCB designs use castellated vias on a 0.100″ x 0.600″ DIP grid. Using castellated vias allows me to design a single large board in the future ( HAB2? ) that could interconnect everything. For HAB1 – I just soldered a whole bunch of cable harnesses – which made the entire design very scalable and flexible.
Simple power resistor for current limiting the LEDs. These boards are sandwiched between the batteries and use excess waste heat to keep the batteries warm.
Battery module containing 2 10,000 mAh 3.7V Lithium Ions, the original solar charger and USB switcher board and a dedicated 5V switcher I purchased that doesn’t turn off ( unlike the one that came with the battery ). A 3rd battery rests underneath the RaspPi A+.
The 5V DC/DC I purchased couple with my voltage and current sensor board and a high side FET board. The DC/DC has a LED that turns on when the battery is above 3.5V. This is used to turn off the FET board when it drops below 3.5V.
The custom electronics on top of the RaspPi A+ plugged into the 2×20 GPIO header on top. During prototyping, the 22 gauge wire was just plugged into the SIP sockets on each board. After pinout was finalized, I soldered all the wires in.
This is what it looks like when you run out of time to design a final interposer PCB and instead connect up all the misc PCBs with wire. Yes – it all worked – even at 103,000 feet (31km).
The top is sealed with the lid. The wires all come out a single slit on the side that is eventually sealed with kapton tape.
Testing the solar panels ability to charge a dead battery.
Testing the RockBLOCK from my house the day it arrived.
Notice the number of failures before the 1st message gets through.
All three modules plus cells with all the cable harnessing to interconnect them. Original mechanical design for HAB1 called for a 1/2″ wooden dowel to run through the center of the sphere – which mandated the three separate tupperware containers into of just a single large container. We ditched the rod and used duct tape instead ( which worked great ). I already had all the cable harnessing done, so kept the three containers. This supported wrapping the battery module and electronics flight computer module in foil for heat retention, alpha particle shielding (rad hard) and also RF noise shielding. RockBLOCK had only the sealed air environment of the tupperware container to retain heat. This can be observed in telemetry thermal readings throughout the flight.
Testing the ballast pump. That little 3V aquarium pump was quite the find. It emptied this entire container in 30 seconds.
Quick road test. GPS works great – RockBLOCK – not so much. I switched to doing road tests in my convertible after this – which was slightly better.
Everything wrapped up and ready. Notice all the foil – especially the long 12″ RaspPi Camera ribbon cable. Giant RF emitter that thing is. RockBLOCK is an amazing little device. The Iridium Satellites are in low Earth orbit at a height of approximately 485 mi (781 km). RockBLOCK on the ground – this little unit, talks to 1 of 66 satellites (some of the time) almost 500 miles away. Getting above the clouds helped tremendously, but 100,000 feet is only 20 miles. Those birds are still way – way up there. RF shielding of all the other electronics was very important. The grey blocks are foam cut up into “cheese blocks” and stuffed in all the extra space. Not so much for insulation, but to keep everything from shifing when the balloon bursts and everything is violently inverted for the parachute deployment. Having all the electronics working during descent was crucial for phoning home landing location.
1st pass at the electronics all in a single sphere ready for launch. My teammates liked the Muppit look. Jon named her Cloey. I ditched the outer foil in the end for fear of blocking Iridium signals. HAB2 is currently in the planning stages. HAB1 is signing off. Thank you HAB1 for all your valiant efforts in getting home.
Thank you for reading my blog. This was quite the 6 week adventure from greenhorn team member to reaching near space and returning to tell about it.
In The Press
Max ( Clive Maxfield ) author of this hilarious book on electronics and also EE Times columnist wrote this article on the HAB1 journey and BML.
Richard Baguley of Hackaday wrote this article on the HAB1 journey.
Lisa Martin of Makezine.com wrote this article on the HAB1 journey.