flōt

When collecting data you have to be very careful to make sure that the data you collect is useful. One of the main problems that you may encounter while trying to tain a network is ambiguity. This is especially true in our case where our system attempts to automatically label all of the images we obtain with the required class. This means that we are fairly reliant on the sensors ability to accurately estimate the data we are recording. Currently, we are using a low-cost ultrasonic sensor to estimate the current distance to a particular obstacle. Given a series of measurements from this sensor we can estimate whether a object exists infront of the data collection setup.
Given the noise present the data is filtered using a Kalman smoother. The smoother is a non-causal filter, this helps to minimize the delay of the signal. Time synchronization is actually one of the fundamental issues when training this kind of system. If images are not properly synchronized with measurements the result is ambiguity. For example, we can imagine a situiation moving from left to right, and on the left side there is empty space and on the right there is a wall. If there is a mis-synrchonization in the data, we may inadvertantly label not just the edge but a solid portion of the wall as free space. This does not even take into account the complexities that image augmentation adds. The augmentation will produce even more examples of images with incorrect labels, that are created in such a way to produce invariance to changes in the images. However, in this example if we were to take random horizontal flips of the image it is likely that we will simply increase the subset of images with ambiguous labels.
It is however inevitable that there will be some bad data in the training set. But it is still a good idea to try to guage how much noise there is present. One easy way to get a rough estimate is to try to overfit a network to the training set. If the network is large enough, it should be able to memorize nearly all the labels to the images. But it is still possible it may get some wrong. Many of these cases are those that are ambiguous. When I looked at our training set when checking this, I noticed several examples of blank walls with alternating labels. These are obvious cases of ambiguity, and they ideally shouldnt even be in the training set. Some caution should be taken with this method, given that a network may simply not be able to learn specific intricacies, and may not necessarily indicate ambiguity and noisy labels. Some additional information on the topic of model capacity can be found in 'Learning Curves: Asymptotic Values and Rate of Convergence'.
The figure shows a cool example of how Fibi's network is able to properly classify tough cases like glass and reflections!

The video above shows a sort of shakey view of something moving around in a room. Thats actually a combination of a network we trained and a heuristic policy. This is representative of some of the work we did during our 4a term when we were setting up our infrastructure and simulation pipeline using AirSim. Many of the ideas that were used during this stage originally came from 'Learning to Fly by Crashing'. The training data that was used in this was collected by randomly sampling trajectories until crashing. A heuristic based labelling system was then applied to generate labels. At a high level, we took the intial part of the trajectories to be of the postive class, and the later part, just before crashing, to be of the negative class. We trained a CNN to attempt to reproduce these classifications. Overall, its reasonable to think of the network to be trying to output the probability of crashing if it were to follow a particular trajectory. After the intial rounds, we go back and collect more data from the failure cases and add them to our training set. This process is quite similar to the DAGGER algorithm.
Now, it may seem to be a bit strange to be generating training data by sampling randomly, but this formulation has a few useful properties. Namely, its very simple to implement. We wanted to be able to test out our training pipeline and software stack as soon as possible. To get it to work we just had to take a several random samples and collect a variety of images. Moreover, there was no human labelling required, one of the key barriers to using a network for a FYDP project. Currently, we are using a modified ResNet-18 pretrained on ImageNet from the PyTorch model zoo. We modified the network to reduce the final classification output to be a single class to match the training labels. This network was selected due to its relatively small size, and ease of accessibility. While there are likely other tasks and datasets which would have provided a better foundation for the network, the features from ImageNet are still an excellent foundation.
In order to reduce the burden of data collection we employ several data augmentation techniques. But first, one of the main issues that we have to keep in mind is to ensure that all changes that we are making do not affect the true label of the image. One of the easier things that can be done is to simply downscale a larger image into the size accepted by the network. This allows for more information to be visible to the network at all times. This comes at the cost of resolution of smaller details within the image. However, in this application, it does not cause much of an issue as the most pertinant parts of the image which may cause collision are assumed to be large. As a caveat, the ability to perform this cropping is largely a function of the Field of View (FoV) of the camera. This is an angular measurement of how far the camera can actually see. Most notably, this limits what a singular camera can actually avoid, as it is limited by what it can see. In general, there is a trade-off between resolution and distortion within the image. Lower FoV cameras will have a much higher resolution of what they can see, and higher FoV cameras will have lower resolution and likely more distortion from the camera lens. This distortion makes the images harder to work with from the perspective of dataset augmentation. In order to ensure that the image distribution of the augmented images and those actually seen are more similar, the augmentation may have to be done on a undistorted image and then redistorted to match. Not replicating such distortions may give hints to the network in a more general case through artifacts.
Looking back, the process of letting Fibi crash randomly in real life is not exactly feasible. We'll talk a bit about how we can change some of these ideas to have more efficient data collection in a later blog post.
The figure below shows an outline of the current network architecture we are using. It was created with the pytorchviz tool.

Fibi is learning how to fly! We are now tuning its control parameters and testing them through our new teleop program. Once it response well to our teleop commands, it will be ready for its final test!
Fibi can freely move about in 3D space through three aerial motion: going up/down, going forward/backward, and turning around. Once Fibi receives velocity command for one, Fibi's embedded controls program will translate these into actuator values for its propellors through PID controllers. Feedback is provided through Fibi's IMU sensor, which tells it about how fast it is currently moving and turning.
We are now improving the controller performance and the control loop rate of which Fibi's controller can provide a stable response for. In recent testing, we have identified points of delay in the control implementation that is affecting it, so we are confident that Fibi can fly even better in the near future! Stay "tuned"!

I wrote earlier about the importance of data for a network. One of the major issues that can occur with data collection is the mislabeling of data. As mentioned earlier this can happen for a variety of reasons ranging from poor sensing data, or from poor timing information. This is not an issue that is exclusive to our project. For example, if you just wanted to classify hotdogs from not hotdogs, this noise would occur if hotdogs were mislabeled by the human expert as not hotdogs, or even if things that are not hotdogs were labelled as hotdogs.
In the context of our project, the latter example is not so much of an issue since computers don't get bored (we automatically label our data). We therefore must focus on optimizing the quality of the first. The tricky part is that the it's not just the quality of the sensors we have to be concerned with, but also the data collecting micro-controller. This is especially true for sensor's who's data must be processed by the micro-controller in real-time. An example of this are some ultra-sonic sensors such as the HC-SR04. You must manually trigger the sensor and wait for the response back. The amount of time taken to receive this response determines the distance to the object. Now this may sound like the perfect application of a busy waiting loop since you want to minimize the amount from response to trigger, but this would have several problems in practice. To name a few, this is an excessive use of CPU time, a waste of battery power and most importantly variability in scheduling may cause the measurements to fluctuate significantly. This is especially true with our selected micro-controller the raspberry pi zero w. It's processing is significantly limited, given an initial implementation of the sensor reading, we noticed a variance of more than 10cm without any load on the system. Not to mention a cpu load of over 25%. There are several ways to fix this, ideally what is needed is access to high accuracy clocks as well as direct access to hardware interrupts. These requirements would circumvent any timing issues through the guarantee that as soon as the signal is received the measurement routine would be run. However, in the case of the raspberry pi, this would require recompiling the kernel to add an additional module. A close second place implementation would be to use interrupts provided through the pigpio daemon. This helps to minimize the response time from the trigger, though has the drawback that the message must be stamped within user-space. Ultimately this may result in some measurement noise. With this change the cpu usage combined between the daemon and the module was decreased by over 10% along with a significant decrease in measurement variance.
To give a bit of context on why timing is important for our project I will give a brief outline of our data collection process. When collecting data from the real world, we simply carry the camera and scan the room. We record images of walls and objects from near and far away. The idea is similar to what was presented in 'Toward Low-Flying Autonomous MAV Trail Navigation using Deep Neural Networks for Environmental Awareness' as in both cases a human user is used to speed up the data collection process. We then use an ultra-sonic sensor to determine the distance to these objects. Afterwards in post-processing we can use a kalman smoothing algorithm to get a better estimate of the distance to the measured objects. With some additional image transformations we can train a network to determine where it is safe to fly. Ultimately, the alignment of measurements and images is key to making this algorithm work.
We can now step back into some of the considerations being made for data collection. Firstly, the standard installation of Raspbian on the pi has several processes wasting valuable time on the cpu, such as cups and lightdm. Their removal significantly reduced idle cpu load, giving us more breathing room in terms of implementation. Another important note is the slow write speeds of the SD card on the raspberry pi. They are rated to be at most 80 Mb/s, which is about 20% of what a normal SSD can achieve. Ultimately, this means that we must be careful with writing to disk. Notably, we noticed that the significant logging from ROS was eating into CPU time, its removal resulted in about a 10% reduction in usage with the system running. Further gains were achieved through coalescing read and write operations to our i2c sensors. Finally, in order to maximize data quality and since we now had room for it, data collection was moved on-board the raspberry pi, instead of streaming it over the network. This significantly reduced the number of observed errors in the data, while having a reasonably small impact on utilization. Storage usually performs much better when performing large sequential writes to memmory, as opposed to the previously mentioned data logging. This helps to explain our ability to even record the data.

So far we have talked about methods for collision avoidance using deep learning. With the right set of labels and inference, it is possible to perform tasks such as collision avoidance in the real world. There is however a fundamental underlying assumption that is needed for this to work. This assumption is that the inference that we make is independent from one another. To put this more concretely, when we make an action decision from images, we assume that it doesn't effect the future images that we'll see. This is certainly true for something like image classification where the prediction that you make on the current image does not have anything to do with the image we will infer in the future. Not only that, we are simply interested in making the correct prediction on the current image at any given time for image classification and these type of tasks are well suited for supervised learning. Our case of collision avoidance differs from this slightly. For most of the robotics tasks including collision avoidance, we are interested in long term performance. This means that the task requires some level of planning to complete the goal effectively. For example, if the goal is to navigate around a maze, there needs to be consideration to how the current action effects performance later in time. So even if going around the long way in a maze might not get you closer to the goal in the immediate time, but it will eventually let you solve the task. The way that current action decision is not independent of the future, makes the problem more challenging. For collision avoidance with deep learning, we are able to avoid some of these issues by predicting an almost general criteria like collision prediction and tuning heuristics to stabilize the algorithm. There is another set of algorithm which tackles the long term planning directly, known as reinforcement learning. We have started to develop reinforcement learning algorithms to perform collision advance as it makes less assumptions and has the potential to perform better eventually. One of the challenges in reinforcement learning is that it requires lots of data and collection of data is more tricky than supervised learning or deep learning methods. Currently we are developing the algorithm in toy simulation environment (Modified MountainCarContinous-v0) and we'll be testing the algorithm in a new simulator that we are trying out called AI2TOHR. The algorithm that we are starting off with is a simple Advantage Actor Critic algorithm.

Fibi's first chassis is manufactured through 3D printing technologies.
The 3D printed chassis was used to build the development platform. The chassis was printed using the STRATASYS J750 printer, as it can print complex shapes accurately, with better resolution than FDM (fused deposit modelling). The printer achieves this by using the proprietary PolyJet technology similar to ink-jet and proprietary materials, to lay minuscule resin deposits/droplets layer by layer that are cured with UV lights as they are laid down.
The team was able to print the chassis with the kind help of the Multi-Scale Additive Manufacturing Lab at the University of Waterloo.
One of the main challenges with this chassis is that the material is not suitable for threads. The threads soon became stripped and were not useful to secure the sensors onto the chassis. The team wil have to explore new materials and/or new designs to overcome these issues.

Last month, we were experimenting with laser cutting acrylic sheets and heat forming it into a simple light weight shield. This month, we experimented with vacuum forming. A thermoplastic sheet is heated to make it flexible, formed over a mold, and then cooled to hold its shape. The mold was 3D printed and this method allowed for intricate features while minimizing the weight.
This enabled us to come up with a whole new design for the blimp chassis. The new chassis was designed with aesthetics and functionality in mind. It has soft features and encloses the electronics effectively. It also includes features for devices mounting, cut-outs, as well as added ribbing for stiffness.

To manufacture the chassis, a male mold was printed using Form2 SLA 3D printer that prints parts by curing resin with a UV laser. Unlike the common FDM printer that usually uses molten thermoplastic, the cured resin mold could withstand the high temperatures of vacuum forming.
The holes and cut-out features on the chassis was indented in the mold and stamped out after vacuum forming using hot cutters.
We were able to use a 0.5mm thin styrene sheet. The new chassis weighs just under 10 grams.

We have been exploring several options to make Fibi's collision prediction faster and more reliable. We are tackling this by moving towards entirely end-to-end trained collision prediction. That is, as opposed to taking multiple crops of the image as previously discussed, we train a model to make predictions from a single forward pass of the network. An example of this is shown in the image above. As before the blue indicates a low probability of collision and the right indicates a high probability. In this particular case, it can be seen that the network correctly predicts that the wall is an obstacle and that it trajectories point towards the other end of the room it is more safe to proceed. Overall, this scheme gives the network the opportunity to learn features that may be lost when simply taking multiple crops. Moreover, the scheme also has the added improvement in computational efficiency, due to the significantly reduced computation within the network. An example network will be shown at the bottom of the page.
One of the more challenging aspects of the data we are collecting is the fact that the labels we receive are sparse. That is, the only information we obtain is the distance to the object directly ahead of us. This data is noisy and the area of measurement increases as it moves away from the robot. That is the sensor has a measurement cone. Ideally, with more expensive sensors it would be possible to generate a label for each of the trajectories that we would like to consider. For example, with a depth camera, free-space and trajectories optimizations can help to determine if a particular trajectory is viable. However, we would like to ensure that we have a low cost solution. Even with this information the network would still have a problem with dealing with multiple correct answers assuming a catagorical output. Namely, if each of the trajectories was given to be a seperate class we would likely run into issues of ambiguity during inference time. This leads us to a required architectural change within the network.
Now with some of the problems defined, we can take a look at some of the solutions we have proposed. Firstly, in terms of the sparse sensor data, we can combat this using image perterbations. We note that the collision prediction we recieve from the sensor is related to the specifically the forwards facing direction. Given that this is the only information we have we must preserve this data label across all of our image perterbations. As before, we can use image crops to emulate being in different positions shifted in space (given more information, or some assumptions we can also rotate the image or scale). After each of the shifts we preserve the original label and mark the original trajectory of the with the label, although, it now is shifted in space (this is similar to what was presented in end-to-end learning for self driving cars). We only have information for that particular trajectory, so we must ensure that we do affect the predictions when we are training. We achieve this, as previously mentioned, through an architectural change.
We must now deal with the problem of having multiple correct answers. Namely at any position any particular trajectory may be correct. As mentioned, this may cause issues with a standard classifier model, using for example a soft-max and cross entropy loss. Instead we can explicitly decompose each of the classifiers, giving each a seperate output. Whenever we backprop, we will only affect the specific outputs meant for that trajectory. This can be achieved in several ways, either through masking of a final classification energies, or by creating multiple classifier heads on the network (this can be seen in the architercural example below). By doing this we completely seperate the tasks of each of the activations. Overall, where we seperate the tasks is an engineering design decision trading off between computation efficiency (closer to the end) and greater representational ability (higher in the network). Without a great enough representational ability the network may be unable to distinguish the different trajectories, while, going to high up will effecitvely result in the same cost as manually cropping. We found a sweet spot in simulation using the final convolutional layers from a resnet-18.
The figure below shows an outline of the new network architecture. It was created with the pytorchviz tool.

Recently we found a really interesting visualization technique developed by NVidia in collaboration with Google Brain and The New York University (NYU), a technique that they suitably chose to call VisualBackProp. While working on our capstone project, due to its research heavy aspect, I have picked a few things on how to efficiently read through research papers to ensure that we're spending more time on the papers that really matter and less so on those that are either not interesting or relevant. An honest side-note in this whole process of determining whether a paper is interesting does come from where the research was conducted, and the "brand" names that are attached to the paper in question. I think it's unavoidable to be completely bias-free, but at the same time those brand names perhaps mean something because those intuitions/companies put out good quality work. In which case, all is fair, and fair is fair.
But aside from the name value of the paper, there are certain things to look out for when efficiently reading through papers. Because realistically there are hundreds of papers relevant to your field that needs to be read, but you also can't afford to treat every paper the same. You got to single out a few favourites. And there are a few steps in finding these gems within the piles of papers that need to be read.
Reading the abstract. While in a business document you will have the executive summary, in a technical paper you will find yourself with the abstract. The abstract should outline the most important parts of the research: the big ideas, the inspiration behind paper and its biggest findings, which in a sense should tell me WHY I should be reading the paper or in another sense tell me WHAT I should be looking for while reading the paper. If, that is, I choose to continue reading deeper into it. This is the reason why the abstract is so important, especially if you're submitting a paper to be thrown into the pile with the rest of the research papers in the digital paper mountains commonly known as arXiv or Google Scholar. Which is also the reason why your grade 6 English teacher made you write what seemed to be an endless amount of summaries, it's just that important. It is true, sometimes the abstract is a bit... how should I say it? Abstract? (pun intended) And you may need to do a very quick scan of the introduction and if provided, look at the visuals within the paper. You shouldn't at this point read the captions or anything, because you still have 99 papers left to scan, and this paper should prove you to that it's worth a deeper read.
It is true that for the VisualBackProp paper, the names "Google" and "NVidia" caught my eye, but by no means was it the whole reason why I decided to dedicate more time to it, and even try to incorporate it into the visualization pipeline. A real-time visualization technique. Now that's impressive. And a theoretical backing of the method to boot. You may know that a lot of the papers produced within the machine learning sphere have little to no theoretical backings as to WHY what was done works, only the empirical data supporting THAT it works. And so those two aspects intrigued me. Ok next step.
First pass. Depending on whether you count the brief skim of the initial screening process as a "first pass", this might be the second pass, but that's just unnecessary details. You now have 10 or 20 papers that you need to go through, maybe less, maybe more, but the process is all the same. It is time now to skim the introduction, take a deeper look at the visuals and go through the sections which explain the meat of the paper. You should avoid the "Related Works" section and anything regarding the theoretical or mathematical heavy. You don't need to dedicate time into reading how the internal mechanisms of the findings works if it's really not work reading into. Now I'll just stop right here to provide an important side-note for the sake of distinction. It is important to distinguish what is shiny and bright versus what is interesting. Not all that glitters is gold, and there are papers that don't have that many visuals but are still very interesting or important or even fundamental to read within the field that you are in. However, a word of comfort is that a lot of times following the bread-trails leads to the candy house (probably not a good analogy because the candy house belongs to an evil witch in the fairy tale). Which is to say, that if a paper catches your eye that you are compelled to read it in much depth, the "Related Works" section and the "Citations" sections will lead you to those crucial papers. The most important papers tend to be cited the most often, so don't sweat it too much. But in the first pass, you just want to know generally what it is that the paper claims to have found. Reading the captions for the visuals and then going to the paragraphs of descriptions tend to help in this process for me. And this should filter out some of the good apples from the bad. Which after you have done this for the rest of the papers, will lead you to the...
Second pass. At this point, there should only be a handful of papers to go through. And at this "depth" of reading you should try to understand it. You still don't need to get your hands dirty in trying to understand the mathematics and theoretical aspects of the paper, but you should get to know what the paper is doing, finding out its inner mechanisms. Actually, I take that back, what I said a few lines ago, which is not to understand the mathematics, but many times there will be formulas that are very important in understanding or just simply something you should already know in delving into the field, these mathematics you should also get a grasp of. I just meant that you don't need to understand the deeper mechanisms such as the proofs and the derivations for the findings, but just enough to know what it does. In the beginning, most papers that reach this stage will continue onto the latter stages as well, but after reading more papers over the course of a short amount of time, you will get a sense of whether something is a bigger deal than what it seemed to be in the initial few passes of the paper. And for those ones, they would fail to move onto the next stage.
Third pass. This is where you would attempt to understand it well. You don't NEED to understand the theoreticals, BUT they will most certainly help you get a deeper understanding of the paper, and reading math comes with practise as with any other language. But another thing that may help in this process is to implement the paper. If it a completely new concept, it might be good to implement it from scratch, if you have the time to do it. But if it heavily builds on previous ground-up implementations that you done, then using a framework to experiment or recreate isn't too bad either. After doing this, interests might be sparked to delve deeper into the field in general. This is where the aforementioned "Related Works" and the "Citation" sections come in handy.
For the VisualBackProp, I applied on our custom ResNet-18-based machine learning model, and processed a few images to see how it performs. A gif was created with those processed images. But what is still really impressive is how quickly these images are processed with the VisualBackProp algorithm even on a GPU-less (or more specifically cuda-incompatible GPU) laptop such as my own.
It was an interesting journey, this capstone project was, and this is one aspect that I have learned and further developed in, in the course of the project's span.

Fibi is getting ready to speak! By leveraging IoT and Amazon Alexa, we are giving Fibi a voice to speak and talk to! Through voice command, you will be able to control Fibi and use its various applications.
Alexa skills are easy to develop and simple to understand. User voice input are catagorized into intents, with slots parsing specific words into variables to be used within the skill. Afterward, a Python server application developed using Flask-Ask can handle these intents and perform specific actions or tasks with them. A good tutorial on how to develop Alexa skill is available here
Currently, Fibi can: take a picture, show a picture, name a picture, and upload a picture to Fibi's Twitter feed. If you're wondering about Fibi's Twitter, you can find it here. Here's a video of it in action:

The server application is running on localhost:5000, which is exposed to the public through ngrok, which is available for free as well as for paid plans.
Fibi's camera feed is made available to the Flask-Ask server via HTTP POST requests from its main Python application, constantly updating it with the latest images from the camera.
The next step will be to integrate the server application with ROS, and allowing the server application to directly control Fibi through motion commands by overriding and publishing on relevant topics.