Live VWAP Calculation with Apache Spark

Volume Weighted Average Price (VWAP) is a popular technical indicator to asses trades that you have made.

I made a simple stock server that randomly updates stock prices and an Apache Spark program to process these results. The code is on github here github/babsher/spark-stocks.

To run the stock server execute the following:

sbt "run-main StockServer 9999 IBM:142 MS:64 AAPL:345"

Then run the VWAP calculation:

sbt "run-main StockStreamingMain"

The key part of the calculation is a simple weighted average implemented by a flatmap and a reduceByKey.

stocks.window(Seconds(30)).
      flatMap(_.map(u => {
        (u.name, (u.vol * u.value, u.vol))
      }))
      .reduceByKey((e1, e2) => (e1._1 + e2._1, e1._2 + e2._2))
      .map{case (stock, (price, vol)) => (stock, price / vol)}

First I used a flatmap to take the RDD of Seq of Stocks and map that to a sequence of the stock symbol and the first weight calculation. Then I used a reduceByKey to sum the numerator and denominator of the average for each stock. The last map calculates the VWAP for each symbol.

If you are feeling adventurous you can set this up in Apache Zeppelin.

Heroku Friendly Dropwizard Mongo

So for a few weeks now I have been using Dropwizard with MongoDB. Two great technologies however the mongo plugin for Dropwizard was missing a configuration option for the mongo URI which is used by Mongo Lab, a monogdb provider for Heroku.

Dropwizard Mongo being an open I decided to update the version of mongo jackson and add the mongo URI configuration option. The results are in my fork. Which you can pull from my Bintray repo.

Here is an example config yaml:

mongoDB:
    dbName: test
    uri: mongodb://db2.example.net:2500

Happy programming!

Analysis of Gingerbreak

For this paper I will be discussing CVE-2011-1823 [NVD2011] which allowed a malicious user to execute arbitrary code and gain root privileges on the Android operating system. This vulnerability  was commonly used to gain root privileges on Gingerbread version of Android until it was patched. The National Vulnerability Database recognizes this CVE-2011-1823 as a class of Numeric Errors (CWE-189). However the more critical problem is that due Due to a lack of Input Validation (CWE-20) the code can be subject to Buffer Errors (CWE-119) and Code Injection (CWE-19). As I will show the lack of validation from a seemingly trustworthy source allows an attacker to compromise the android operating system.

The vulnerability beings in DirectVolume.h where the number of partitions on the system is stored as an signed integer [GOOGLE2015-1] line 36. Later in DirectVolume.cpp an array called mPartMinors is initialized with -1s [GOOGLE2015-2] line 42.

Excerpt from DriectVolumn.h

Later when the program receives an event that a partition is added it stores a value from the event to an index in this array that is specified by the event. The code in the second excerpt from DriectDrive.cpp only checks that the part_num variable is not greater than mDiskNumParts which does not prevent part_num from being negative. Thus if the PARTN parameter of the message is negative the value of the minor variable will be written to an arbitrary place in memory. The mistake was trusting the input from the socket which even though it is local can still be accessed by unprivileged programs.

The events come from a local socket connection, called a PF_NETLINK socket, that is supposed to be only used by the operating system for special events. However, access to this socket has no authorization protection or authentication. Therefore, a malicious user can inject specifically crafted packets that use a negative offset in this array to write an arbitrary values to memory. The code trusts the data is coming from an outside source it should still be validated.that the operating system is the only one writing messages, however even if this is true since the data is coming from an outside source it should still be validated.

Excerpt from DirectDrive.cpp

The vulnerability was addressed in 2 ways; first the check on line 189 of DirectDrive.cpp was updated to ensure part_num was greater than 0 [GOOGLE2015-3] . This will ensure that line 200 will not write data to memory outside of the array. The second update was to the PF_NETLINK socket used to communicate with the kernel. The socket was updated to require authentication and authorization to write to the socket, thus preventing malicious users from writing to the socket.

References

[NVD2011]

http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2011-1823

[GOOGLE2015-1] https://android.googlesource.com/platform/system/vold/+/2dfe297ec47559dbe2297a72bea71cf515c03797/DirectVolume.h

[GOOGLE2015-2] https://android.googlesource.com/platform/system/vold/+/2dfe297ec47559dbe2297a72bea71cf515c03797/DirectVolume.cpp

[GOOGLE2015-3] https://android.googlesource.com/platform/system/vold/+/f3d3ce5e53ab7928f4c292c183c417a1bd051151%5E%21/#F0