<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0">
   <channel>
      <title>projectlog</title>
      <link>http://www.provolot.com/projectlog/</link>
      <description></description>
      <language>en</language>
      <copyright>Copyright 2008</copyright>
      <lastBuildDate>Mon, 13 Aug 2007 03:06:51 -0500</lastBuildDate>
      <generator>http://www.sixapart.com/movabletype/</generator>
      <docs>http://blogs.law.harvard.edu/tech/rss</docs> 

      
      <item>
         <title>Perl script for upload</title>
         <description><![CDATA[Geez, it's been a while. I'm ashamed.

I started writing a Perl script to process input files, connect to the PostgreSQL databse using DBI, query the tract info, and to store entries (including census tract info) back into the database. So far, the script does everything but record entries.

Some key points of the script:
<pre class=code>my $tract_sth = $dbh->prepare("SELECT t.tract FROM cshape t WHERE ST_Contains(t.geom, GeomFromText(?)) = TRUE");</pre>
Prepare, uh, prepares the query for execution later so that a specific point can be inserted where the question mark is.

<pre class=code>
$lon =~ m/^(\d{3})(\d{2}\.\d{4})/; ### grab degrees and minutes
$deg = $1; $min = $2;
$londeg = $deg + $min / 60.0; ## do the conversion
$londeg =~ s/^(\d*\.\d{8}).*/$1/; ## limit decimals to eight numbers
if($line_segments[6] =~ m/W/) { $londeg = -$londeg; }
</pre>
This segment of the code grabs the degrees and minutes, divides minutes by 60 and adds to degrees to convert to decimal degres, then using Regex, limits decimals to eight numbers. Finally, it parses a segment of the GPRMC data to see if the value should be positive or negative. This all is only possible because of Perl's built-in ability to analyze strings as both numbers and strings on-the-fly.

Now, all we need is for time, date, and census tract info to be stored in a new table. After that's done, the Processing app will query the database and find all results between certain time/dates. (The time/dates will be indexed?) A list of census tracts will have values that increment for each second of time spent in a tract. The Processing application will get the first and second query, figure out the difference in time, and increment the counter for the census tract corresponding to that amount of time. Then, the application will get the second and third... and so on and so forth. After that all, the Processing application will get census info for the tracts (also stored in a database table!) and calculate corresponding data.

Handling transportation will come later, but will consist of looking for sudden gaps of time over a threshold, and dividing the change in distance by the change of time. Then, we will able to get the average location for certain points in time. 


]]></description>
         <link>http://www.provolot.com/projectlog/2007/08/perl_script_for_upload.html</link>
         <guid>http://www.provolot.com/projectlog/2007/08/perl_script_for_upload.html</guid>
        
        
         <pubDate>Mon, 13 Aug 2007 03:06:51 -0500</pubDate>
      </item>
      
      <item>
         <title>success</title>
         <description><![CDATA[I compiled geos -- standard './configure', 'make', 'make install'. I then compiled PostGIS again: 
<pre class=code>LDFLAGS=-lstdc++ ./configure --with-geos=/usr/local/bin/geos-config</pre>
which was pretty quick, surprisingly. 'make', 'make install', and then done! Bingo!

But the query wouldn't work, wouldn't return anything. After messing around for a while, I realized that TRACT was lowercase, and Contains should be ST_Contains. It still didn't work, until I realized that ST_Contains returns a boolean value, and tried comparing to 'TRUE' instead of '1':

<pre class=code>SELECT t.tract FROM cshape t WHERE ST_Contains(t.geom, GeomFromText('Point( -73.965331 40.806129)')) = TRUE;</pre>

The address values are different this time, because the previous address was right on the boundary of two census tracts and I just wanted to be sure. I got:

<pre class=code> tract 
-------
 0199</pre>

Awesome! <a href="http://factfinder.census.gov/home/saff/main.html?_lang=en">American FactFinder</a> verifies this. 

GPS->Census Tract info correlation success! Now for the data/information stuff.

]]></description>
         <link>http://www.provolot.com/projectlog/2007/07/success.html</link>
         <guid>http://www.provolot.com/projectlog/2007/07/success.html</guid>
        
        
         <pubDate>Sat, 21 Jul 2007 19:38:37 -0500</pubDate>
      </item>
      
      <item>
         <title>PostGIS continued, but...</title>
         <description><![CDATA[So things were going faster than expected. I installed PostGIS, then in the source directory fount a tool called shp2pgsql. Using the command
<pre class=code>shp2pgsql -w -g geom tr36_d00 cshape >
tr36_d00.pgresimport</pre>
I turned the shapefiles into batch files that are executable by PostgreSQL. The -w turns the format into wkt format (well-known-text), which I just liked because it was readable. I had trouble with settings, and fudged around this area for a while.

I then created the standard OpenGIS tables as dictated in <a href=http://postgis.refractions.net/docs/ch04.html#id2692440>this page</a>: the SPATIAL_REF_SYS and GEOMETRY_COLUMNS tables. It seems that they're mandatory. I then  imported the file that I had previously created:
<pre class=code>psql -Upostgres -d census_shapes -f tr36_d00.pgresimport</pre>
postgres is the user, census_shapes is the database, and tr36_d00.pgresimport is the import file. This worked, and all the entries started importing! Hurrah. After that, I tried querying:

<pre class=code> select AsText(geom) from cshape limit 1;</pre>
And I got an output that looked like:
<pre class=code>MULTIPOLYGON(((-74.922368 44.960301,-74.917243 44.963743,-74.902403 44.947712,-74.901045 44.94551,-74.90028 44.944099,-74.898672 44.941034,-74.898415 44.940851,-74.898125 44.940097,-74.894169 44.935157,-74.893477 44.93416,-74.893223 44.933621,-74.89314 44.933076,-74.89314 44.932367,-74.890155 44.921848,-74.888541 44.916816,-74.887571 44.91426,-74.885153 44.911527,-74.88254 44.908058,-74.881842 44.898385,-74.886792 44.904059,-74.888457 44.905923,-74.892535 44.910287,-74.895901 44.913879,-74.89628 44.914283,-74.900616 44.918999,-74.90068 44.919068,-74.908067 44.926972,-74.908321 44.927241,-74.911828 44.930953,-74.912202 44.931349,-74.914032 44.933287,-74.919177 44.938664,-74.919808 44.939322,-74.920282 44.939845,-74.922369 44.942046,-74.924504 44.944292,-74.930209 44.950191,-74.931129 44.951419,-74.932078 44.95242,-74.934773 44.955263,-74.924425 44.959972,-74.922368 44.960301)))</pre>
Great! And now for the final test: the query that I ran earlier on MySQL and didn't work --
<pre class=code>SELECT t.TRACT
FROM cshape t
WHERE Contains(t.geom, GeomFromText('Point(-73.963956 40.808502)')) = 1;
</pre>
And I get:
<pre class=code>ERROR:  contains:: operation not implemented - compile PostGIS with JTS or GEOS support</pre>

Uhh, what?

It turns out that I didn't have GEOS support compiled. GEOS (<a href="http://geos.refractions.net/">Geometry Engine Open Source</a>) is a C/C++ port of the JTS (<a href="http://www.jump-project.org/project.php?PID=JTS&SID=OVER">JTS Topological Suite</a>), which handles topological calculations with polygons/lines/etc, spatial analysis, etc. Of course, this means that JTS and GEOS provide the capability to answer my crucial question: "Is point X inside shape Y"?

So -- I downloaded the newest version of GEOS, and am compiling it now. I believe (according to <a href="http://postgis.refractions.net/docs/ch02.html#PGInstall">this page</a>) that I need to recompile PostGIS by using the options:
<pre class=code>LDFLAGS=-lstdc++
./configure --with-geos=PATH</pre>

I'm soo close.


<hr />
<b>update:</b>
I finally fixed it <a href=http://www.provolot.com/projectlog/2007/07/success.html>here: success</a>. ]]></description>
         <link>http://www.provolot.com/projectlog/2007/07/postgis_continued_but.html</link>
         <guid>http://www.provolot.com/projectlog/2007/07/postgis_continued_but.html</guid>
        
        
         <pubDate>Sat, 21 Jul 2007 16:28:27 -0500</pubDate>
      </item>
      
      <item>
         <title>PostgreSQL &amp; PostGIS! </title>
         <description><![CDATA[I installed readline with fink, and then compiled PostgreSQL from source (as helpfully explained in <a href="http://developer.apple.com/internet/opensource/postgres.html">this page</a>.)
Compiling PostGIS from source..
http://postgis.refractions.net/docs/ch02.html
To Be Continued...


<hr />
<b>update:</b>
I fixed this eventually; check the solution process <a href=http://www.provolot.com/projectlog/2007/07/postgis-continued-but.html>here: PostGIS continued, but...</a>, and finally fixed it <a href=http://www.provolot.com/projectlog/2007/07/success.html>here: success</a>. ]]></description>
         <link>http://www.provolot.com/projectlog/2007/07/postgresql_postgis.html</link>
         <guid>http://www.provolot.com/projectlog/2007/07/postgresql_postgis.html</guid>
        
        
         <pubDate>Thu, 19 Jul 2007 16:55:20 -0500</pubDate>
      </item>
      
      <item>
         <title>YEAH! and NO!</title>
         <description><![CDATA[Awesome! and Oh No!

So I finally got it working. The <a href="http://jcole.us/software/libmygis/">libmygis</a> library in the previous post had a tool called 'mysqlgisimport', that I used to import my shapefiles with. I had to tell it not to use a PRJ file (whatever that is) since I didn't have one, and outputted the results into a temporary file:
<pre class=code>./mysqlgisimport --no-prj tr36_d00  -o foo</pre>
Notice that the shapefiles' extensions aren't written. After that, I took foo, and inputted it into mysql to run mysql in batch mode:
<pre class=code>mysql -upb -p < foo</pre>
('pb' is my username, so -upb significates that the user is pb, and -p makes it ask for a password.)
And that worked! All the files went into the mysql database, into a table called tr36_d00! What worked this time vs. using the shp2mysql.pl tool is that a column holding geometry values was created. To test the creation of these geometries, I tried querying:
<pre class=code>select AsText(tr36_d00.geo) FROM tr36_d00 limit 1;</pre>
and got entries like
<pre class=code>POLYGON((-73.519309 45.006876,-73.452584 45.008767,-73.437371 45.009198,-73.411169 45.009655,-73.3883 45.010053,-73.367388 45.010417,-73.357332 45.010592,-73.352228 45.010681,-73.343124 45.01084,-73.350188 44.994304,-73.353355 44.990259,-73.353429 44.990165,-73.354633 44.987352,-73.354112 44.984062,-73.353716 44.982959,-73.352886 44.980644,-73.350218 44.976222,-73.345974 44.971764,-73.34474 44.970468,-73.339451 44.966433,-73.338734 44.965886,-73.338243 44.96475,-73.337906 44.960541,-73.339603 44.94337,-73.338482 44.924112,-73.338939 44.918194,-73.338979 44.917681,-73.339142 44.917447,-73.339534 44.916885,-73.396083 44.911148,-73.396627 44.913411,-73.450909 44.908153,-73.486053 44.904716,-73.507148 44.911338,-73.521172 44.911536,-73.520173 44.941891,-73.519375 44.97616,-73.519309 45.006876))</pre>

Awesome! So I tried testing to see if it could correlate between GPS data and census tract. Using a geocoder, I got the lat/long of 116th st and Broadway in NYC (<a href="http://maps.google.com/maps?q=40.8080,+-73.963956">40.8080, -73.963956</a>). Using <a href="http://dev.mysql.com/tech-resources/articles/4.1/gis-with-mysql.html">these</a> examples as help, I constructed this query and ran it:
<pre class=code>
SELECT t.TRACT
FROM tr36_d00 t 
WHERE Contains(t.geo, GeomFromText('Point(-73.963956 40.808502)')) = 1;
</pre>
and with my tired eyes I saw in slowly-dawning-hope-growing-turning-into-joy:
<pre class=code>
+-------+
| TRACT |
+-------+
| 0315  | 
| 0205  | 
| 0203  | 
+-------+</pre>

YAY HOORAY AWESOME OH BOY Oh boy oh oh....

...unfortunately, as you can see, there are three TRACT answers, which doesn't make sense, as the census tracts don't overlap. Alas, the function Contains, which checks to see if an object contains another objects, is not accurate. As <a href="http://www.mysql.org/doc/refman/5.0/en/functions-that-test-spatial-relationships-between-geometries.html">this page of documentation</a> says, it instead returns the value from the MBRContains function. And the MBRContains function doesn't actually check, it checks to see if the <i>smallest bounding box</i> of an object contains the <i>smallest bounding box</i> of another's.

Damn! Because Manhattan doesn't run exactly north-south, but its census tracts do, the bounding rectangles (which presumably have vertical and horizontal lines) are probably woefully inaccurate:
<img alt="censusblockmbrexample.gif" src="http://www.provolot.com/projectlog/censusblockmbrexample.gif" width="378" height="380" />
Here are the minimal bounding rectangles; there's a third one you can't see because the census block is thin and tall, and so its MBR is too large. The black dot is the location of the lat/long coordinates on the map. Since the dot is within both MBRs, the function treats it as being contained, and that's why we get three census tracts back -- because the coordinates are within the MBRs of three census tracts: 315, 205, 203.

GAH! so what now? Some guy wrote a function to check for true 'contains' by getting the coordinates of the census tract, drawing a line from the desired point outwards, and counting how many times the line crosses the boundaries of the census tract? If the number of times is odd, its contained; if not, then it isn't. Described <a href="http://local.wasp.uwa.edu.au/~pbourke/geometry/insidepoly/">here</a> and implemented in PHP(why?) <a href="http://www.mysql.org/doc/refman/5.0/en/functions-that-test-spatial-relationships-between-geometries.html">here</a>.

But no, that's sloppy. I don't want to do that! I think I'll end up installing <a href="http://postgis.refractions.net/">PostGIS</a> and using that instead, which unlike MySQL has excellent spatial/GIS support -- or so I hear. 

All that time spent on MySQL! Wasted! AHHHH!

 
<hr />
<b>update:</b>
I fixed this eventually; check the solution process <a href=http://www.provolot.com/projectlog/2007/07/postgresql-postgis.html>here: PostgreSQL & PostGIS! </a>, <a href=http://www.provolot.com/projectlog/2007/07/postgis-continued-but.html>here: PostGIS continued, but...</a>, and finally fixed it <a href=http://www.provolot.com/projectlog/2007/07/success.html>here: success</a>. ]]></description>
         <link>http://www.provolot.com/projectlog/2007/07/yeah_and_no.html</link>
         <guid>http://www.provolot.com/projectlog/2007/07/yeah_and_no.html</guid>
        
        
         <pubDate>Thu, 19 Jul 2007 01:14:51 -0500</pubDate>
      </item>
      
      <item>
         <title>MySQL Spatial abilities = bad</title>
         <description><![CDATA[Holy shit this stuff is so convoluted. Apparently MySQL doesn't have good spatial/GIS support, compared to PostGIS/PostgreSQL. On top of that, nobody seems to have some sort of documentation -- there are all these tools and scripts lying around that nobody knows how to use. There's a 140+ page pdf lying around explaining some standard of representation of sorts. 

I'm sure it shouldn't be this difficult. I'll try to use a library called <a href="http://jcole.us/software/libmygis/">libmygis</a>, and if that doesn't work out, then I'm just going to switch to PostGIS, dammit..


<hr />
<b>update:</b>
I fixed this eventually; check the solution process <a href=http://www.provolot.com/projectlog/2007/07/yeah-and-no.html>here: YEAH! and NO!</a>, <a href=http://www.provolot.com/projectlog/2007/07/postgresql-postgis.html>here: PostgreSQL & PostGIS! </a>, <a href=http://www.provolot.com/projectlog/2007/07/postgis-continued-but.html>here: PostGIS continued, but...</a>, and finally fixed it <a href=http://www.provolot.com/projectlog/2007/07/success.html>here: success</a>. ]]></description>
         <link>http://www.provolot.com/projectlog/2007/07/mysql_spatial_abilities_bad.html</link>
         <guid>http://www.provolot.com/projectlog/2007/07/mysql_spatial_abilities_bad.html</guid>
        
        
         <pubDate>Wed, 18 Jul 2007 23:26:54 -0500</pubDate>
      </item>
      
      <item>
         <title>DBD::mysql fixed!</title>
         <description><![CDATA[Finally fixed the whole DBD::mysql shenaningans by creating root-owned simlinks from /usr/local/mysql/lib/mysql (where the system was looking) to /usr/local/mysql/lib (where it was) or so. 

Also, the script wouldn't work, giving me 'cannot connect' errors, until I went and used the GRANT command to give the user permission. Now it works!

<hr />
<b>update:</b>
I fixed this eventually; check the solution process <a href=http://www.provolot.com/projectlog/2007/07/mysql-spatial-abilities-bad.html>here: MySQL Spatial abilities = bad</a>, <a href=http://www.provolot.com/projectlog/2007/07/yeah-and-no.html>here: YEAH! and NO!</a>, <a href=http://www.provolot.com/projectlog/2007/07/postgresql-postgis.html>here: PostgreSQL & PostGIS! </a>, <a href=http://www.provolot.com/projectlog/2007/07/postgis-continued-but.html>here: PostGIS continued, but...</a>, and finally fixed it <a href=http://www.provolot.com/projectlog/2007/07/success.html>here: success</a>. ]]></description>
         <link>http://www.provolot.com/projectlog/2007/07/dbdmysql_fixed.html</link>
         <guid>http://www.provolot.com/projectlog/2007/07/dbdmysql_fixed.html</guid>
        
        
         <pubDate>Mon, 16 Jul 2007 21:42:48 -0500</pubDate>
      </item>
      
      <item>
         <title>mysql, perl packages, fink, shp2mysql</title>
         <description><![CDATA[<p>To import shp files into a MySQL database, someone made a perl script called <a href="http://hips.ifi.uio.no/confluence/display/REP/WebGIS+Plan+Autumn+2005">shp2mysql</a>. Unfortunately, when I ran this, it told me that my computer required several different modules. After running around, I decided to install <a href="http://finkproject.org/">Fink</a>, a sort of apt-get thing for OSX that enables you to use Unix software on OSX. I then installed a few Perl libs, but realized that Fink didn't have all of them available. <br />
</p><p>
I then tried to use CPAN, which I suppose is like Fink/apt-get, but just for Perl libraries. CPAN's ftp wouldn't work however, which may be a combination between using active vs. passive ftp. To fix this, I set <pre class="code">&rt; export FTP_PASSIVE=1</pre>in the bash shell of a NON-ROOT user, then did<pre class=code>sudo perl -MCPAN -e 'install Bundle::DBD'</pre>
It still doesn't work, however. Here are the error messages:
</p>
<pre class=code>
Can't load '/Library/Perl/5.8.6/darwin-thread-multi- 2level/auto/DBD/mysql/mysql.bundle' for module DBD::mysql: dlopen(/Library/Perl/5.8.6/darwin-thread-multi- 2level/auto/DBD/mysql/mysql.bundle, 1): Library not loaded: /usr/local/mysql/lib/mysql/libmysqlclient.15.dylib
  Referenced from: /Library/Perl/5.8.6/darwin-thread-multi-  2level/auto/DBD/mysql/mysql.bundle
  Reason: image not found at /System/Library/ Perl/5.8.6/darwin-thread-multi- 2level/DynaLoader.pm line 230.
 at -e line 1
Compilation failed in require at -e line 1.
BEGIN failed--compilation aborted at -e line 1.
</pre>
<p>
Gah.</p>

<hr />
<b>update:</b>
I fixed this eventually; check the solution process <a href=http://www.provolot.com/projectlog/2007/07/dbdmysql-fixed.html>here: 
DBD::mysql fixed!</a>, <a href=http://www.provolot.com/projectlog/2007/07/mysql-spatial-abilities-bad.html>here: MySQL Spatial abilities = bad</a>, <a href=http://www.provolot.com/projectlog/2007/07/yeah-and-no.html>here: YEAH! and NO!</a>, <a href=http://www.provolot.com/projectlog/2007/07/postgresql-postgis.html>here: PostgreSQL & PostGIS! </a>, <a href=http://www.provolot.com/projectlog/2007/07/postgis-continued-but.html>here: PostGIS continued, but...</a>, and finally fixed it <a href=http://www.provolot.com/projectlog/2007/07/success.html>here: success</a>. ]]></description>
         <link>http://www.provolot.com/projectlog/2007/07/mysql_perl_packages_fink_shp2m.html</link>
         <guid>http://www.provolot.com/projectlog/2007/07/mysql_perl_packages_fink_shp2m.html</guid>
        
        
         <pubDate>Mon, 16 Jul 2007 03:53:42 -0500</pubDate>
      </item>
      
      <item>
         <title>spatial database extensions</title>
         <description><![CDATA[I installed mysql on osx yesterday, and finished setting root passwords for all hostnames. Now i need to utilize the <a href="http://dev.mysql.com/doc/refman/5.1/en/spatial-extensions.html">spatial extensions</a> and correlate them with the census block shapes (from the TIGER/Line data). 

Next to do: 
- Load data into mysql. 
- Figure out how to do queries -- x contains y?
- Write Perl or Processing script to process queries. This <a href="http://www.bezier.de/mysql/">Processing-MySql library</a> may work.]]></description>
         <link>http://www.provolot.com/projectlog/2007/07/spatial_database_extensions.html</link>
         <guid>http://www.provolot.com/projectlog/2007/07/spatial_database_extensions.html</guid>
        
        
         <pubDate>Fri, 13 Jul 2007 14:59:53 -0500</pubDate>
      </item>
      
      <item>
         <title>Boundaries</title>
         <description><![CDATA[Okay, slowly getting there.

There are 'cartographic boundary files' downloadable <a href="http://www.census.gov/geo/www/cob/metadata.html">here</a>. This is the <a href="http://avce00.maptools.org/docs/v7_e00_cover.html#ARC">technical documentation</a>.

Incredible. Seems to be a giant collection of nodes, later creating boundaries. I'm not sure how to process this..]]></description>
         <link>http://www.provolot.com/projectlog/2007/07/boundaries.html</link>
         <guid>http://www.provolot.com/projectlog/2007/07/boundaries.html</guid>
        
        
         <pubDate>Mon, 09 Jul 2007 17:05:32 -0500</pubDate>
      </item>
      
      <item>
         <title>TIGER</title>
         <description><![CDATA[I haven't been posting often.

Currently I'm trying to interface <a href="http://www.census.gov/geo/www/tiger/tgrcd108/tgr108cd.html">TIGER/Line</a> files with GPS data to translate lat/long coordinates into street data, and consequently census blocks.

I may have to use <a href="http://search.cpan.org/~sderle/Geo-Coder-US/">this</a> code later.]]></description>
         <link>http://www.provolot.com/projectlog/2007/07/tiger.html</link>
         <guid>http://www.provolot.com/projectlog/2007/07/tiger.html</guid>
        
        
         <pubDate>Mon, 09 Jul 2007 15:18:52 -0500</pubDate>
      </item>
      
      <item>
         <title>GPS logging parsing</title>
         <description><![CDATA[<p>Okay, this code parses the data for a proper message, then gets only the lines starting with that message. This was relatively messy, especially in C/C++. Note that the function filterForMessage doesn't pass a value really, but instead just modifies a global variable. I tried messing with pointers, but I don't think Arduino's code can handle them.</p>

<pre class="code">


#define rxPin 7
#define txPin 8
#define ledPin 13

//definitions for 4800 baud communication
#define bit4800Delay 188 
#define halfBit4800Delay 94 

//The type of message we want. Edit this for other forms of GPS data.
char wantedMessage[7] = "$GPRMC";

// thisMessage is a global variable -- I didn't want to deal with returning
// strings, as properly object-oriented it would be. 
char thisMessage[100];
boolean ledPinOn;

void setup()  {
  // define pin modes for tx, rx, led pins:
  pinMode(rxPin, INPUT);
  pinMode(txPin, OUTPUT);
  pinMode(ledPin, OUTPUT);
  digitalWrite(txPin,  HIGH); //set transmit pin to high
  digitalWrite(ledPin,  HIGH); //turn on debug led  
  ledPinOn = true;
  
  // set the data rate for the SoftwareSerial port
  Serial.begin(4800); //4800 is the baud rate for EM306
}

int SWread()
{
  byte val = 0;
  
  while (digitalRead(rxPin));
  //wait for start bit
  
  
  if (digitalRead(rxPin) == LOW) {  
  //okay, so now that it's low
    delayMicroseconds(halfBit4800Delay);
    //wait for a tiny bit, then get data
    for (int offset = 0; offset < 8; offset++) {
     //iterate through bit mask.
     //Or, in other words, each bit is sent after a bit4800delay,
     //from MSB to LSB. get the first most significant bit, shift it left, etc.
     delayMicroseconds(bit4800Delay);
     val |= digitalRead(rxPin) << offset; 
    }
    //wait for stop bit + extra
    delayMicroseconds(bit4800Delay); 
    delayMicroseconds(bit4800Delay);
    return val;
  }

}

boolean filterForMessage()
{
  int tempVal;
  int counter;
  char tempString[100]; //max string length is 100.
  
  tempVal = SWread(); //convert byte from GPS to character
  if(tempVal == '$') //if byte is a start of a new line 
  {
    counter = 1;
    tempString[0] = '$'; //add the dollar sign
    
    //while we're reading input, and the input isn't a newline (10 is ascii newline)
    while((tempVal = SWread()) && (tempVal != 10)) {
      tempString[counter++] = tempVal; //add current character to temporary string
    }
    tempString[counter] = '\0'; //end string with null char
    
    //now, do we want this line? let's compare the first 6 chars with wantedMessage.
    if(strncmp(wantedMessage, tempString, 6) == 0) {
      //if we do want this line, then copy the message into the global variable
      //so we can 'pass' it to the callee of this function.
      strcpy(thisMessage, tempString); 
      return true;
    }
    else
      return false;
     
  }
 
}
        
      
void loop() {
  
  //if our message matches our wanted message, print it out!
  if(filterForMessage() == true) {
    Serial.println(thisMessage); 
  
    // toggle an LED so that we can see that it's working
    // the LED will go on with every other proper message match
    if(ledPinOn == true) ledPinOn = false;
    else ledPinOn = true;
    digitalWrite(ledPin,  ledPinOn);
  }

}
</pre>

<p>So, let's look at the data. An example line:<br />
$GPRMC,083739.000,A,3922.17xx,N,07640.02xx,W,0.38,32.39,210507,,*26<br />
Clearly I'm not <a href="http://maps.google.com/maps?f=q&hl=en&q=39+22'+17%22N+76+40'+02%22W&ie=UTF8&ll=39.367989,-76.671631&spn=0.004247,0.008669&t=h&z=17&om=1">here </a> right now or anything.</p>


<p>The <a href="http://www.sparkfun.com/datasheets/GPS/NMEA%20Reference%20Manual1.pdf">reference manual</a> for the NMEA standard details the RMC format in page 16. <a href="http://woodshole.er.usgs.gov/operations/sfmapping/gps.htm">This page</a> has it as well.
</p>
<p>A bit of data parsing in Processing led to this code. String manipulation in Java is more awkward than I remembered -- I can't seem to be able to use 'split' well.</p>

<pre class="code">
 boolean parseRMCData(String raw) {
    
    Date tempDate;
    
    String [] values = raw.split(","); //split values around comma
    
    if(values[2].equals("V")) return false; //status is invalid!
    
    
    //convert values into date. GAH! there should be a better way,
    //but split wasn't working properly.
    int h; int m; int s; int y; int mon; int d;
    h = parseInt(values[1].substring(0, 2));
    m = parseInt(values[1].substring(2, 4));
    s = parseInt(values[1].substring(4, 6));
    d = parseInt(values[9].substring(0, 2));
    mon = parseInt(values[9].substring(2, 4));
    y = parseInt(values[9].substring(4, 6)) + 100;
    
    timeDate = new Date(y, mon, d, h, m, s);
    //100 + y is done because y is two-digits, and the Date class
    //automatically adds 1900 to the year value. Adding 100 corrects this all.
    
   
    //// CONFUSING TIMEZONE CONVERSION COMING RIGHT UP
    
    // configure a SimpleDateFormat to recognize our certain format
    SimpleDateFormat sdf = new SimpleDateFormat("y M d H m s");
    
    // configure a time zone that is EST and with dst built in.
    
    // Base GMT offset: -5:00
    // DST starts:      at 2:00am in standard time
    //                  on the second Sunday in March
    // DST ends:        at 2:00am in daylight time
    //                  on the first Sunday in November
    // Save:            1 hour
    SimpleTimeZone stz = new SimpleTimeZone(-18000000,
                              "EST",
                              Calendar.MARCH, 2, -Calendar.SUNDAY,
                              7200000,
                              Calendar.NOVEMBER, 1, -Calendar.SUNDAY,
                              7200000,
                              3600000);
    
    // configure our SimpleDateFormat to recieve GMT input
    sdf.setTimeZone(TimeZone.getTimeZone("GMT"));
    
    // make our SimpleDateFormat parse our date as GMT
    try {
      tempDate = sdf.parse(y + " " + mon + " " + d + " " + h + " " + m + " " + s);
    }
    catch (Exception e) {
      tempDate = new Date();
    }
    
    // convert SimpleDateFormat to EST with EDT
    sdf.setTimeZone(stz);
    
    // change display pattern of our format 
    sdf.applyPattern("E M/W/y H:m:s z");
    
    // convert the date to EST with EDT, and in our certain pattern
   print(sdf.format(tempDate));

   print("  |  Lat : " + values[3] + values[4] + "  |  Long : " + values[5] + values[6]);
   
   float  vel = Float.parseFloat(values[7]) * 1.852;
   println("  |  Velocity : " + vel + " km/h in the direction " + values[8] + " degrees from north");
  
   return true; //status is valid, of course
    
  }
</pre>]]></description>
         <link>http://www.provolot.com/projectlog/2007/05/okay_this_code_parses_the.html</link>
         <guid>http://www.provolot.com/projectlog/2007/05/okay_this_code_parses_the.html</guid>
        
        
         <pubDate>Mon, 21 May 2007 03:51:11 -0500</pubDate>
      </item>
      
      <item>
         <title>WORKS!1</title>
         <description><![CDATA[<p>holy smokes, it works!
GPS EM-406 to Arduino:
<pre class="code">


#define rxPin 7
#define txPin 8
#define ledPin 13

#define bit4800Delay 188 
#define halfBit4800Delay 94 
//definitions for 4800 baud communication

byte val;
boolean ledPinOn;

void setup()  {
  // define pin modes for tx, rx, led pins:
  pinMode(rxPin, INPUT);
  pinMode(txPin, OUTPUT);
  pinMode(ledPin, OUTPUT);
  digitalWrite(txPin,  HIGH); //set transmit pin to high
  digitalWrite(ledPin,  HIGH); //turn on debug led  
  ledPinOn = true;
  
  // set the data rate for the SoftwareSerial port
  Serial.begin(4800); //4800 is the baud rate for EM306
}

int SWread()
{
  byte val = 0;
  
  while (digitalRead(rxPin));
  //wait for start bit
  
  
  if (digitalRead(rxPin) == LOW) {
    
    
    delayMicroseconds(halfBit4800Delay);
    for (int offset = 0; offset < 8; offset++) {
     delayMicroseconds(bit4800Delay);
     val |= digitalRead(rxPin) << offset; 
     //iterate through bit mask.
     //Or, in other words, each bit is sent after a bit4800delay,
     //from MSB to LSB. get the first most significant bit, shift it left, etc.
    }
    //wait for stop bit + extra
    delayMicroseconds(bit4800Delay); 
    delayMicroseconds(bit4800Delay);
    return val;
  }

}

void loop() {
  Serial.print(SWread(), BYTE); 
  if(ledPinOn == true) ledPinOn = false;
  else ledPinOn = true;
  digitalWrite(ledPin,  ledPinOn);
 
  // toggle an LED just so you see the thing's alive.  
  // this LED will go on with every OTHER character received:
  

}
</pre>
</p>
<p>output (but with no spaces between commas):<br />
$GPGGA,062032.000,3922.1678,N,07640.0290,W,1,06,1.5,132.3,M,-33.6,M,,0000*6D
$GPGSA,A,3,28,11,17,08,27,19,,,,,,,2.4,1.5,1.9*31
$GPRMC,062032.000,A,3922.1678,N,07640.0290,W,0.16,134.81,210507,,*11
$GPGGA,062033.000,3922.1678,N,07640.0290,W,1,06,1.5,132.3,M,-33.6,M,,0000*6C
$GPGSA,A,3,28,11,17,08,27,19,,,,,,,2.4,1.5,1.9*31
$GPGSV,3,1,09,28,70,332,41,11,57,067,43,08,49,205,38,17,43,269,26*7D
$GPGSV,3,2,09,27,29,187,29,20,14,122,,19,09,063,24,26,05,289,*7C
$GPGSV,3,3,09,29,04,281,*44
$GPRMC,062033.000,A,3922.1678,N,07640.0290,W,0.18,139.48,210507,,*16
</p>
<p>Now, to parse the $GPRMC -- the recommended minimum specific gnss data...</p>
]]></description>
         <link>http://www.provolot.com/projectlog/2007/05/works1.html</link>
         <guid>http://www.provolot.com/projectlog/2007/05/works1.html</guid>
        
        
          <category domain="http://www.sixapart.com/ns/types#tag">arduino</category>
        
          <category domain="http://www.sixapart.com/ns/types#tag">gps</category>
        
         <pubDate>Mon, 21 May 2007 02:21:08 -0500</pubDate>
      </item>
      
      <item>
         <title>things connected, not working.</title>
         <description><![CDATA[<p><img alt="before_after_gpsconnector.jpg" src="http://www.provolot.com/projectlog/2007/05/21/before_after_gpsconnector.jpg" width="400" height="325" /><br />
before:wires stripped
after: soldered to 22 guage wire, then wrapped around with scotch tape. I don't have electrical tape or shrink tubing on me now, nor the automotive operational skills to get any...
</p><p>
<img alt="gps_breadboard.jpg" src="http://www.provolot.com/projectlog/2007/05/21/gps_breadboard.jpg" width="400" height="300" /><br />
connector and arduino plugged into the breadboard.
</p>
<p>Things aren't working! I tried connecting the GPS module to the connector. The red led lit up, then after a while started blinking, which according to the <a href="http://www.sparkfun.com/datasheets/GPS/EM-406%20Product_Guide1.pdf">datasheet</a> means that the position is fixed. Yay! That's great.</p>
<p>As for communicating with the EM406 chip via serial, it's not working. The arduino is supposed to communicate through serial, at 4800 baud, but I can't find the protocol online to request data. I swear that I found a series of timing diagrams on a pdf somewhere with the signals that have to go to rx and tx pins, but I can't find it now. </p>
<p><a href="http://itp.nyu.edu/~bp432/blug/archives/2007/03/gps_sparkfun_module_gprmc.html">Benedetta Simeonidis</a> was also working on interfacing the Arduino and the EM406. She seems to have got it working, but her code doesn't work for me. I tried my own code, which doesn't work either. Here's the code anyways.</p>
<pre class="code">


#define rxPin 7
#define txPin 8
#define ledPin 13

#define bit4800Delay 188 
#define halfBit4800Delay 94 
//definitions for 4800 baud communication

byte val;
boolean ledPinOn;

void setup()  {
  // define pin modes for tx, rx, led pins:
  pinMode(rxPin, INPUT);
  pinMode(txPin, OUTPUT);
  pinMode(ledPin, OUTPUT);
  digitalWrite(txPin,  HIGH); //set transmit pin to high
  digitalWrite(ledPin,  HIGH); //turn on debug led  
  ledPinOn = true;
  
  // set the data rate for the SoftwareSerial port
  Serial.begin(4800); //4800 is the baud rate for EM306
}

int SWread()
{
  byte val = 0;
  Serial.println("waiting for start bit");
  
  while (digitalRead(rxPin))
  {
    Serial.println(digitalRead(rxPin), BIN);
  }
  //wait for start bit
  
  Serial.println("done waiting");
  
  
  if (digitalRead(rxPin) == LOW) {
    
    Serial.println("rxPin is low");
    
    delayMicroseconds(halfBit4800Delay);
    for (int offset = 0; offset < 8; offset++) {
     delayMicroseconds(bit4800Delay);
     val |= digitalRead(rxPin) << offset; 
     //iterate through bit mask.
     //Or, in other words, each bit is sent after a bit4800delay,
     //from MSB to LSB. get the first most significant bit, shift it left, etc.
    }
    //wait for stop bit + extra
    delayMicroseconds(bit4800Delay); 
    delayMicroseconds(bit4800Delay);
    Serial.println("got data");
    return val;
  }
  Serial.println("rxPin is not low");
}

void loop() {
  Serial.println("started loop");
  Serial.print(SWread(), BYTE); 
  if(ledPinOn == true) ledPinOn = false;
  else ledPinOn = true;
  digitalWrite(ledPin,  ledPinOn);
 
  // toggle an LED just so you see the thing's alive.  
  // this LED will go on with every OTHER character received:
  

}
</pre>
<p>
The point at which things get stuck is at 
<pre class="code">
  while (digitalRead(rxPin));
  //wait for start bit
</pre>
where the protocol for communicating with the GPS chip seems to involve waiting for the receiving rx pin to be low briefly, then to initiate data input. This seems strange.</p>

<p>Where can I find that datasheet??</p> 

<hr />
<b>update</b>: <a href=http://www.provolot.com/projectlog/2007/05/works1.html><u>it works</u></a>.
]]></description>
         <link>http://www.provolot.com/projectlog/2007/05/things_connected_not_working.html</link>
         <guid>http://www.provolot.com/projectlog/2007/05/things_connected_not_working.html</guid>
        
        
         <pubDate>Mon, 21 May 2007 01:56:18 -0500</pubDate>
      </item>
      
      <item>
         <title>more stuff purchased</title>
         <description><![CDATA[<img alt="spools_breadboard.jpg" src="http://www.provolot.com/projectlog/2007/05/20/spools_breadboard.jpg" width="400" height="300" />
Time to start interfacing the EM406 and the Arduino with serial...]]></description>
         <link>http://www.provolot.com/projectlog/2007/05/more_stuff_purchased.html</link>
         <guid>http://www.provolot.com/projectlog/2007/05/more_stuff_purchased.html</guid>
        
        
          <category domain="http://www.sixapart.com/ns/types#tag">purchased</category>
        
         <pubDate>Sun, 20 May 2007 18:02:59 -0500</pubDate>
      </item>
      
   </channel>
</rss>
