Presentation-Aid Armband with IMU, EMG Sensor and Bluetooth for Free-Hand Writing and Hand Gesture Recognition

Purpose –The study aimed to improve the presenter’s capability to give a presentation in a hands-free manner. It covered the design of a wearable armband that uses electromyography (EMG), Inertial Measurement Unit (IMU), and Bluetooth wireless technology. Also, it covered the development of presentation software for Windows operating system. Method –The study employed the common elements of engineering design process which includes problem identification, requirement analysis, design solution, implementation, and testing. Prototyping approach was used to design the armband to ensure that it can properly gather raw EMG and IMU signals. Object-Oriented Programming (OOP) was used to develop the presentation software so it could interpret the received signals into common hand gestures and facilitate free handwriting on air. Results – Raw EMG and IMU signals could be transmitted from the armband to the attached computer via Bluetooth successfully. The developed software was able to interpret the received signals from the armband and perform the corresponding computer navigation commands. It is able to recognize and translate dynamic gestures into an equivalent handwriting. Furthermore, it is able to read different file types such as Portable Document File (PDF), PowerPoint, Word document and audio-video media files. Conclusion – The angular velocity and linear acceleration data from arm movement along with the electromyographic data from forearm muscle contraction could be successfully used to implement hands-free navigation and free-hand writing on air. Thus, the integration of IMU with the EMG signals for dynamic hand gestures and free-handwriting work accordingly. Recommendations – To increase the number of recognizable gestures, additional EMG sensors must be placed in more strategic positions. Handwriting functionality could be improved with the use of additional handwriting recognition algorithm. Research Implications


INTRODUCTION
Presentation aids are very useful to deliver a better speech as it helps audience to understand what the speaker is trying to convey.Presentation aids that are normally used include actual objects, photographs, audio-videos, diagrams, and charts (ChangingMinds, 2010).One of the most popular presentation-aids is the Multimedia Projector.Survey shows that in 2002, 91 percent of the educators in the United States are using Multimedia Projector for their presentations (The Journal, 2002).Majority of the presentation files are presented in PowerPoint, Word, and PDF format.Thus, Microsoft PowerPoint is now probably the most-widely used form of visual aid (University of Leicester, 2004).Over the past decades, various presentation tools have been developed to improve the manner of delivering presentation.These include overhead projector, LCD projector, interactive whiteboard, and interactive projector, to name a few.
Nowadays, presentation devices are integrated with wireless technology to improve the presenter's mobility and capability to freely express ideas about a topic (Top Ten Reviews, 2016).Research proved that gestures increase the value of a message by 60 percent (Science of People, 2015).Unfortunately, available presentation-aid devices limit the hand gestures of the user because of the need to hold a hardware.Writing back and forth to the board on top of an existing presentation is another thing to consider.With the advent of Human Computer Interaction (HCI), wearable computing devices are now starting to emerge from the realm of science fiction into actuality.The rapidly expanding market of wearable technology is indicative of the near limitless potential for changing the relationship between users and information context(s) (Gandy, Baker, & Zeagler, 2017).
Although the concept of making presentation is not quite new, the need to effectively communicate and to properly convey the information a speaker intends to impart has been part our human nature and will continue to evolve.Imagine a device that could be worn on the arm instead of being held by the hand that will make it possible for a presenter to navigate and to write on thin air without holding anything.To be able to do the same thing without restricting one's hand freedom and flexibility might be viewed as a luxury.But with the increasing demand for wearable technology, a hands-free presentation toolkit with handwriting capability is thought to be a necessity and is but one step away from becoming a reality.

OBJECTIVES
The general objective of this study is to develop a Bluetooth-enabled wearable presentation toolkit for hands-free navigation and wireless handwriting using electromyography and inertial measurement unit.
In line with this, the project aimed to achieve the following specific objectives:  to create a wearable armband that can transmit coded signals to a host computer for common navigation gestures, and  to develop a presentation software that o interprets the received signal from the armband and they make possible basic computer navigation and handwriting functionality; o reads presentation files such as audio, video, Word, Excel, PowerPoint, and PDF; and o saves handwriting and modified presentation into a new file.

The Importance of Computer Aided Presentation Tools
In the primary school level, computer-aided educational environment has been observed to be more effective than that of conventional education in terms of increased learning (Kose, 2009).Since the early '90s up until today, LCD projector and PowerPoint presentation have been the most widely-accepted presentation aid.Accompanied with either laptop or desktop computers, it allows the lecture to be projected on a much larger screen for the entire audience to visually grasp what the speaker is trying to convey.
Across different disciplines, speakers want to improve the way the message could be delivered more effectively.But it is not only about using computer with visually appealing presentation that has a total effect on the learning process.In the study conducted in India that compares the traditional methods of teaching such as blackboard and chalk with lecture delivery using projectors and PowerPoint presentation, results showed that majority of the students preferred blackboard and chalk over projector and PowerPoint presentation (Singh, Gupta, & Srivastava, 2016).As was found from the interview, the main preference was not in the visual presentation but with the teacher's interaction with the students.
In the early days, although PowerPoint presentation greatly improves the manner of presenting, the interaction with the audience and the presentation itself was quite limited.The speaker continually had to walk in front of the projector screen to access the computer keyboard or mouse to change the slide.The speaker was forced to stay close to the computer to be able to navigate and interact with the display file being presented.This scenario somewhat limited the speaker's ability to interact with the audience both physically as well as verbally.Upon the introduction of wireless presenters which is also known as remote presenter pens, many educators and lecturers found it to be of great help.It is for this reason wireless presenters allow the lecturer to walk around the room, and engage with the audience, and still navigate the presentation slide efficiently.Most modern wireless presenters have an attached laser pointer that could be used to direct the viewers' attention to a specific part of the projected image.
However, in the field of engineering and other similar disciplines, mathematical symbols, and diagrammatic forms are the key elements.These have been traditionally taught using oral and handwritten approaches also known as chalk-talk (Wilson & Maclaren, 2013).The use of PowerPoint presentation in most classroom-based settings for engineering domain leads to mathematics presented as a solution rather than a process.Many teaching environments no longer provide suitable whiteboard spaces to sketch and to write an engineering solution on.In addition, the lecturer could become constrained by the requirement to use the digital projection and PowerPoint.The use of pen-enabled Tablet PCs allows a return to a dynamic handwritten approach and allows drawing and sketching in vivid color (Wilson & Maclaren, 2013).
Along with the LCD projector, interactive whiteboards are starting to gain popularity and continuous to improve throughout the years.Although it comes with a price, this interactive whiteboard was said to have many advantages for teachers that require manipulation of objects in real time like in Physics classes (Stoica, Paragina, Paragina, Miron, & Jipa, 2011).The use of LCD projected arouses the students' interest and motivate them to participate.While in more developed countries, there has been an extensive investment by governments and individual schools in interactive whiteboard technology with the assumption that their use in education will impact positively on learners' achievements.Other developing countries raise concerns about the lack of ICT literacy for both teachers and learners and the cost of the interactive whiteboard technology (Slay, Sieborger, & Hodgkinson-Williams, 2008).

Human Computer Interaction: Hand Gestures and Handwriting
The use of a physical controller like mouse, keyboard for human computer interaction hinders natural interface as there is a strong barrier between the user and the computer.A study was conducted to develop a system that uses a camera to track both static and dynamic hand gestures.The said system did not use any marker and was able to translate the detected gestures into computer actions like opening a website and launching an application such as VLC and Microsoft PowerPoint (Haria, Subramanian, Asokkumar, Poddar, & Nayak, 2017).Despite the efforts of the scientific community, the hand gesture recognition issue is still unresolved and attempts are still being made to provide a fullynatural interface.One study employed Microsoft's Kinect sensor technology and concluded that their proposed shape recognition solution proved to be viable for realtime implementations and has shown to have a good recognition success ratio despite the low resolution provided by the depth frame.The validation methods proved to be reliable in evaluating a template based approach and leaves no doubts of the good performance of the solution.As a recommendation, more validations are required for testing different hand shapes and a library so that further conclusions could be made and threshold parameters are better tuned (Cardoso, Delgado, & Barata, 2015).
As other methodologies for hand gestures are continuously being studied, pressure sensors and other non-invasive interface for hand gesture recognition has not been completely evaluated.Jiang, Merhi, Xiao and Menon (2017) made an examination regarding the performance of hand gesture classification using Force Myography (FMG) and surface Electromyography (sEMG).Using an FMG band prototype to perform 3 sets of 48 hand gestures, the study revealed that FMG band achieved classification accuracies as good as the high quality commercially available (Jiang et al., 2017).
Muscle contraction sensors like Electromyography are commonly known for its medical use and application.EMG signals are generated by the skeletal muscle.Intramuscular EMG signals are detected using wires or needles inserted into the muscles, while surface EMG signals are detected using a pair of electrodes or more complex array of multiple electrodes attached to the surface above the muscle on the skin.EMG is also used in many medical laboratories in giving diagnostic tests used to evaluate the body's nerve and muscle function and assesses a potential neurological disease (John Hopkins Hospital, 2017).Since then, electromyographic signals associated with handwriting have received little attention.In 2009, Linderman, Lebedev and Erlichman (2009) conducted a method in which EMG signals generated by hand and forearm muscles during handwriting activity are reliably translated into both algorithm-generated handwriting traces and font characters using decoding algorithms.Results demonstrated the feasibility of recreating handwriting solely from EMG signals (Linderman et al., 2009).

Design Phase
The study employed the common elements of engineering design process.The armband requirement specifications are as follows: it must be operated by a 6-volt battery; it must be adjustable for different arm sizes; it should be capable of wirelessly transmitting coded signals within a 10-meter radius; it should have the capability to detect recognizable forearm-muscle contractions; it should capture inertial motion of arm movements; it should be usable for either left or right arm operations.The presentation software requirement specifications are as follows: it should allow user to calibrate the paired device; it should be able to read presentation files including Word document, PowerPoint, PDF, and audio-video media files; it must be able to translate electrical signals into common hand gestures and allow free-hand writing; it must be able to save handwritings into a new file.
Figure 1 illustrates the level 1 diagram of the presentation-aid armband.It was designed to accept input signals from the user's muscle activities and arm inertial motion via electromyography sensor and inertial measurement unit which includes accelerometer and gyroscope.The microcontroller handles the received data and generates an electrical signal to be transmitted via Bluetooth to a host computer for further interpretation.

Implementation Phase
The device prototype was constructed using the following components: MyoWare muscle sensor for attaching biomedical sensor pads directly to the board to produce raw EMG output signals; Arduino MPU 6050 sensor for collecting angular velocity and linear acceleration data from arm movement to produce raw IMU signals; Arduino pro minimicrocontroller (gizDuino mini) for processing raw EMG and IMU signals and for sending it to the host computer via Bluetooth dongle.Figure 2 shows the aforementioned armband hardware components.The operation of the presentation-aid toolkit starts from the user's muscle activity as an input to the armband.Analog signals are then generated and processed by the microcontroller.The device establishes a connection to the host computer using Bluetooth wireless technology.Once connected, continuous data transmission would take place.The digital signals received by the software will then be interpreted for hand gestures as either navigation command or free-hand writing.Figure 3 shows the process flowchart of the armband and the presentation software.

Testing and Verification Phase
This phase covered the individual component testing, integration testing, and acceptance testing.Component testing involved the following: electrical signals produced by EMG sensor having a unique function; IMU sensors generating correct x, y, and z axes values by position; and Bluetooth dongle transmitting and receiving data between host and slave device.Integration testing involved microcontroller's functional capability to coordinate with the EMG and IMU sensors, and to be able to control Bluetooth communication device to transmit data correctly to the host computer's presentation software.Acceptance testing involved the presentation software's capability to detect valid dynamic gestures from the armband with minimal latency and continuous operation within the range of a 10-meter radius.

Presentation of the Developed System
Figure 4 shows the developed armband prototype.It consists of EMG sensor, IMU sensor, Gizduino Mini ATMega328, Bluetooth module, and 6v Battery.Three holes are present for the replaceable electrodes to be attached.LED power indicator is also present on top of the armband.Closing of fist gesture turns on the device.
After initializing, it needs to be connected to the host computer via Bluetooth communication. Figure 5 shows the armband calibration via the presentation software installed on the host computer.Each image is highlighted as the corresponding gesture is performed by the user.Flex operating point, represented by green progress bar, could be adjusted to suit the user's need.There are seven (7) basic hand gestures the prototype could recognize and it could perform 17 different operations in the software.The presentation software allows the combination of these gestures to perform specific commands.Furthermore, selecting the draw menu option allows the user to sketch and write anything on top of an existing presentation.Pen size could be adjusted and pen color can be changed.Figure 8 shows the floating pen tool with color picker and pen size located at the upper right side of the presentation screen.Any writings and sketches made on top of the currently displayed presentation page could be saved without overwriting the existing presentation file.A new image file will then be created once the save menu option is selected.Finally, selecting the disable armband deactivates the armband from controlling the presentation, which gives the user freedom to do any gesture without affecting the current presentation being displayed.

Testing and Verification Result
The functional test and verification test results are as follows:  The 6-volt rechargeable battery could power-up the armband for 6 hours of continues usage. Accurate EMG signals could only be acquired properly when placing the sensor pads in the right position where muscle activity is prominent. The presentation software responds to the hand gestures transmitted by the armband and recognizes it with minimal latency.It requires at least 10 to 15 seconds synchronization time after the successful Bluetooth connection in order to experience real-time operation. Free-hand writing mode shows a slightly noticeable cursor movement, more likely because of IMU sensor sensitivity.An initial 5-second delay due to data buffering is observed before real-time free-hand writing can be achieved.

RECOMMENDATIONS
This study recommends the following:  to further improve the wearable armband presentation toolkit, additional EMG sensors placed in a more strategic location must be studied,  to further improve hand gesture recognition and free-hand writing functionality, additional machine learning library with handwriting recognition algorithm must be incorporated,  to improve the device transmission latency, 2.4 GHz Wireless technology may be used instead of Bluetooth technology,  to improve the functionality of the presentation software to support different file types, open-source programming languages such as Python and Java may be considered, and  to improve the saving functionality of the presentation software, other destination file types such as PDF and Word document must be used instead of just flattened image file.

IMPLICATIONS
Flexibility and mobility are important factors in delivering any presentation without limiting the wearer's hand freedom to perform any action needed to properly convey desired message.Modern presentation tools like interactive whiteboard or interactive projector offers a lot of promising functionality.The use of an add-on wearable device to an existing accustomed laptop and LCD projector for lecture and presentation purposes will benefit anyone.Small and large organizations, especially academic institutions, would benefit from the presentation-aid armband because the device will give them the experience of having a modern presentation tool without being compelled to invest so much.

Figure 1 .
Figure 1.Level 1 diagram of the Presentation-Aid Armband

Figure 3 .
Figure 3. Process Flowchart of the Armband and the Presentation Software

Figure 6 .
Figure 6.Presentation Software Floating Menu Option Figure7shows a sample video file being played.The media player screen also displays a floating tool at the bottom, for media navigation such as play, pause, stop, and volume control.

Figure 7 .
Figure 7. Presentation Software Custom Media Player

Figure 8 .
Figure 8. Presentation Software Free-Hand Writing Mode Wireless wearable presentation toolkit with hand writing capability could be constructed with the use of Bluetooth, IMU and EMG sensors.Placing the muscle sensor electrodes on top of the forearm muscle improves the raw EMG signals produced. Combination of EMG and IMU signals could be translated into a recognizable hand gestures.Mouse event allows free-hand writing functionality but with minimal latency and slight cursor movement. Visual C# along with the .NET framework allow Windows Form application to read various presentation and media files. Presentation files could be overwritten without destroying the original file by creating additional layers of images on top of the presentation page.A separate image file is necessary to leave the original presentation file untouched.