Happy New Year to all PHPers!

The yearly PHP community summaries are already appearing throughout the blogosphere. Every year they just get better and better and it becomes clear that the PHP community's accomplishments outshine those of the previous year.

PHP for many years has been the leader in Web development. In the past couple of years though this leadership has become even more noticable. Evans Data Corporation has estimated that over 4.5M developers worldwide use PHP today. Not only is that an impressive number on its own, but when you dig into the numbers you can see that in 2007 PHP is expected to surpass Java in areas like Europe; and that includes all permutations of Java usage including in environments where PHP doesn't attempt to be relevant such as the desktop.

2006 has clearly been a very exciting year for PHP. It comes at a time where the Web, Web applications and Web services are changing the way we communicate and do business. A large amount of the flagship Web sites are running PHP including Yahoo!, Facebook, Flickr (now Yahoo!) and many more. The blogosphere is heavily dependent on PHP applications such as Wordpress, Drupal and others (most of which provide far more value than just blogging). The PHP community is clearly a movement of great sharing and collaboration. Just by looking at the number of available scripts on Hotscripts.com one can see not only the sheer size of PHP but also how that compares to other technologies.

PHP and the community have evolved significantly in 2006. Here are some of the highlights in no particular order:

• PHP 5 adoption tipped in 2006. While there have been various opinions on the matter, judging by the metrics that I have personally seen which are based not on installed base but on what people are actually developing in, I have no doubt that around 75% of the PHP community today develops on PHP 5. The big change seems to have happened with the release of PHP 5.1.
  

• The development of PHP 6 has made tremendous progress with the main new feature being native Unicode support. We are going into 2007 with a pretty solid core code base which already has most of the Unicode work completed. The community is also making great progress in getting all extensions up to speed. Have no doubt, this is a big effort and I believe the release cycle for PHP 6 will be especially long due to the huge amount of changes that were made. The feedback loop with the user base will also be especially important. That said, don't expect another Perl 6 (any news on that?). We've always been very pragmatic in making progress and getting major releases out the door.

• 2006 is no doubt the year of the PHP frameworks. With the Web growing up, increasing migrations of traditional applications into the Web browser, and with the facing of a new wave of three-tier architectures (DB, PHP, Javascript) frameworks are an important piece in making it easier and more efficient for people to deliver such applications. While we now face the challenges of three tier architectures with Javascript in the browser being the third-tier I don't envy J2EE developers who have to now get used to a four-tier architecture (DB, Java Application Server, Servlet/JSP, Javascript in the browser). We launched the Zend Framework project back in October 2005 in order to provide the necessary building blocks and best practices to the PHP community. While already quite a few frameworks had existed at the time, I have no doubt that the launch of this project has significantly increased the interest and discussions regarding frameworks in the PHP community.
  

• On the commercial side, 2006 was a great follow-up year to the 2005 announcements with IBM's Information Management and Oracle's Database teams. In 2006, we announced two important partnerships:

• The first with IBM's System i division (AS/400) where PHP will be used to Web-enable the platform. There are a large number of computers and applications in the System i community, applications are mainly written in RPG & Cobol using the DB2/400 database, and Web enablement has always been a challenge for that community. Due to the ease-of-use of PHP, and with the help of language and DB2/400 connectivity APIs we're providing, we believe that PHP will be a great success on the platform. We have already seen great interest in 2006, but I believe 2007 will be the year where we will see the first set of solutions being rolled out on the platform.

• The second partnership we announced was with Microsoft. More info on one of my earlier blog entries. In the past few months we've already made huge progress in getting PHP to run better on Windows and have contributed those patches to the PHP source tree. In addition, Microsoft has been making very good progress on developing a FastCGI solution for their IIS Web Server. All pieces will be shipping in 2007 and will be made available to all.

• On March 1st we launched the Zend Developer Zone. The creation of this new site was also part of the PHP Collaboration Project which is a three project initiative to further the art of PHP. The goal of this site is to deliver best practices and information which will be valuable to PHP professionals.
  

• The third piece in the PHP collaboration project was the launch of the PDT (PHP Development Tools) project at Eclipse. The goal of this project is to provide the Eclipse eco-system with the necessary framework (not to be confused with Zend Framework) for supporting the PHP language enabling the development of professional tools that leverage this project. Many companies have development tools based on Eclipse including IBM, SAP and BEA and will now have an easier time in supporting PHP if they wish to do so.

• Last but not least are the Zend products. We've had many important releases this year including Zend Studio 5.5, our industrial-strength runtime environment Zend Platform 2.2, Zend Core for i5/OS, Zend Guard 4, numerous Zend Core for IBM & Oracle refreshes, and more... Coming early in 2007 is Zend Platform 3 which is already in beta and has some very cool new features like Job Queue and High Availability session clustering, and some major Zend Core revs including one which includes a lot of the Windows work we have recently been doing.

All of these efforts and many others which have happened within the PHP community and Zend will guarantee an even more exciting and successful 2007.

On a side note, when a technology becomes as ubiquitous like PHP, skeptics always appear. The mantra of dynamic languages is to get the job done, quickly. That is the big commonality between their communities and is what's driving them deep into mainstream use. While I believe PHP will continue to lead the pack, no doubt that its success has opened the door for other languages to follow. I believe the big winners coming out of 2007 will be dynamic languages which is the reason why many of the big players have started to invest in them including IBM (who as usual were way ahead of the pack), Oracle, BEA, Sun & Microsoft. I even think that Redhat (JBoss) might start embracing dynamic languages in 2007, depending on whether the Redhat leadership itself gives its blessing. In addition, it will be interesting how open-sourcing Java will play out in 2007 but many believe that it's too little too late. It seems that these efforts are always being made a bit too late after the respective technology already starts to loose steam. Solaris had the same issue although I think it has benefited from having been open-sourced. However, it will not likely be able to reclaim world dominance despite it being a very good and proven OS. End of side-tracking... :)

In addition I believe you will see the following happening in 2007:

• PHP 5 will continue to expand and mature in parallel to the work on PHP 6.

• The community will make significant progress on PHP 6. I believe we will get a lot of helpful feedback once we get a preview out the door.

• Zend Framework 1.0 will ship H1/2007.

• The native PHP MySQL driver might be released (?).

• Windows Server "Longhorn" will ship with FastCGI support.

• The community will continue to invest on all levels in dealing with the security challenges Web infrastructure faces including best practices, tools, and PHP enhancements and fixes.
  

• There are still some vendors out there who are doing little to actively engage the PHP community and work on better interoperability between their solutions and PHP. I believe we'll have at least one more in the mix.

• We will see a significant improvement in tooling and best practices around PHP & Ajax development.

• I demoed a proof-of-concept of what we called phpBlox in my php|tek 2006 keynote. We are continuing to work on this and some more concrete work should appear in 2007.

If you got this far you really deserve a very happy and restful new year!
Thanks to everyone in the PHP community and its supporters for making 2006 such a successful year. I apologize ahead of time for anything significant which I mistakenly left out.
We have a busy and exciting year ahead of us!

2007 Talks

2007 is already looking like a great conference year. In the upcoming year, I've decided to diversify the conferences I attend in order to meet new people, address new communities and in general increase PHP's exposure in new places.

As always, the period before and after the holiday season is quiet, but towards the end of February things pick up rapidly. Here's what I've already got on my schedule:

AjaxWorld Conference & Expo - "How and Why PHP Makes a Successful Back End for AJAX" - March 19-21, 2006, New York

SD West 2007 - "Modern Web Applications & PHP" - March 19-23, 2007

MySQL Conference & Expo - "The State of PHP and MySQL" - April 23-26, 2007

IBM Webcast - "Vista Powering Rich Internet Applications with Zend Core for IBM" - Feb 28, 2007

So it's definitely going to be an exciting beginning to the conference season. It'll be my first time talking at AjaxWorld and SD West and I'm looking forward to meeting those new audiences and hearing how they use or intend to use PHP.

Happy Holidays to everyone!

Zend Framework 0.6 Posted!

In the spirit of the Zend Framework, we did another anti-release this weekend. Why anti-release? In the commercial world doing a product release on the weekend is a big no-no, but the framework team seems to always converge towards a weekend; and we believe there's no reason to delay a release in order to have "better" timing. In fact, releasing on a weekend has some advantages. There are many community members who have some spare time to provide quick feedback if there has been a mess up, and that makes it easier to "tweak" things towards the beginning of the week. Also, and in contrast, we have found the Zend Framework community to be very quick to pick up such releases, even on the weekend. I think that truly shows the vibrant community and interest that this project has created.

Enough blabbing about anti-releases and onto the real news...
Zend Framework 0.6 is a hugely important release for us. First of all, as the jump in version number reflects, this release really signals how much the Zend Framework has matured over the past few months. Some important signs of this maturity is the very much improved controller architecture which has been based on the experiences and feedback of the community who helped us understand some of the shortcomings in the initial versions, it includes better (and hopefully final) designs for ACL and Session handling, a new Authentication module, a great amount of i18n work which will hopefully move to core for 0.7, and a huge amount of additional features, improvements and bug fixes.

Also, I'm very excited to mention that Zend Framework 0.6 now features the official PHP 5 Google Gdata module. Google have been generous enough to donate this component to the Zend Framework, and we are including it both in the framework itself and making it available separately for download at http://framework.zend.com/download/gdata.
As one of our goals for the framework is to have good Web Services support, whether for generic Web Services such as REST, SOAP and XML-RPC or concrete Web Services like Gdata, Amazon or Flickr, having this as part of the framework is very much in sync with our goal.

So where do we go from here? We have posted a roadmap which details our plans. We're aiming to do at least monthly preview release in order to stay on track with this roadmap. We have also posted a more detailed Google Spreadsheet on the status of the various pieces. While we still expect some slight changes in the roadmap, taking into account community feedback and contributions, we believe that the final first version of the framework won't be far away from this roadmap. As already mentioned in the past, we are going to be focusing almost solely on this roadmap until we ship a 1.0 in order to ensure that we release a good solid first version. Once that happens, it will be easier for us to go back and deal with the overwhelming amount of good proposals we've received from the community. Naturally there will be some flexibility for proposals which are straightforward and have a strong code base and maintainer (such as the ones listed in the "Opportunistic improvements" section of the roadmap) but those will be more the exception than the rule.

For a more graphical view of what we expect Zend Framework 1.0 to be (give or take a few details) please see the following chart which I believe articulates it quite well:



I'd like to thank everyone who has helped make this important release happen. Foremost the community who has made significant contributions and provided invaluable feedback, the official Zend team that has worked late evenings and weekends on this release, and the unofficial Zend team which are Zenders who contribute in their spare time in addition to their existing full-time jobs.

No time to waste... Tomorrow we start working on 0.7...
Happy Holidays to everyone!

Making PHP on Windows work!

Today we announced a partnership with Microsoft. The goal of this technical collaboration is to make PHP on Windows a first class citizen. It has always been our goal to make PHP run well on any platform, and this initiative reinforces that. Both companies have also observed huge interest both from the community and our respective customers bases in good PHP support on Windows, and therefore, collaborating on making this happen is a no-brainer for both.

This is actually not the first time where PHP and Windows have crossed paths. Five years ago, Microsoft invited the PHP development team to come to Redmond and spend a week in their labs to optimize PHP on Windows. Zeev, Shane Caraveo and myself represented the PHP team. We actually made quite a few significant improvements but since that time, there hasn't been much focus on making PHP run well on Windows. Also, five years ago the team that invited us Redmond didn't have very much of a corporate mandate so that effort was never really followed through.

Fortunately, things have changed. There are now people at Redmond who understand that supporting interoperability between open-source software and the Windows platform will have benefits to their customer base. 4.5M PHP developers are hard to ignore, especially when so many PHP developers actually do their development on Windows.

For Zend it has always been important to make PHP as widely applicable as possible. Just this year we announced support for the System I platform (AS/400). With the wide reach of Windows and having significant customer demand for Windows support, it makes a lot of sense to work with Microsoft on getting better integration between the two solutions.

One of the things I'm really excited about is that Microsoft was really willing to change their internal plans in order to better support PHP. Although they've had FastCGI requests from their user-base implementing it never actually happened. It also wasn't on the plans for IIS 7. However, recognizing that good PHP support really required a first-class FastCGI implementation, they managed to be exceptionally agile for a large company and change IIS 7 plans to include FastCGI support. Today they are releasing a preview release of this FastCGI implementation which in our labs tests performs very well and they continue to improve it. This preview release will support both IIS 7 (Windows Vista and Windows Server "Longhorn") and will also support Windows Server 2003 and Windows XP. Not only is this good news for many dynamic languages out there that support FastCGI but it's also a good sign that they have embraced an open de-facto standard as opposed to creating their own "FastCGI" standard.

The second piece in making PHP work well on Windows is PHP itself. After spending a lot of time on benchmarks it was clear that PHP performance on Windows was way behind other platforms. The trivial improvement would of course be a thread-safe build (gives about a 40% performance boost). But there were several other issues which have significantly impaired PHP performance on Windows. Up-to-date we already have over a dozen of improvements that we've made which make PHP run much faster on Windows, whether using better APIs, better algorithms or better utilize the available tools. We have released a technology preview version of Zend Core which include these improvements and people are welcome to play around with it. In the coming weeks we will share these patches with the PHP community and work towards including such improvements into PHP. For more information on the IIS FastCGI component visit the IIS Web site.

In addition, we will do continuous engineering to make sure that PHP continues to work well on Windows, application compatibility issues are met, and make sure that we continue to serve the community and business who are interested in good Windows support. Getting PHP to perform well and be stable on the Windows platform was the highest priority for this collaboration. It's the basis which any further collaboration would build on. Over the coming months we will continue to investigate other ways to strengthen PHP's interoperability with the Windows platform and try and resolve other issues which might arise. We've already received various requests for better Active Directory integration, SQL support, URL rewriting solution, and more. Besides the obvious need for making PHP work on Windows, there are many other improvements which PHP developers could take advantage of.

I very much welcome the Microsoft initiative to invest in getting PHP to run well on their platform. For Zend, working with Microsoft is in line with our collaborative strategy to further PHP usage and allow it to play well in heterogeneous environments. Whether it's Linux, AIX, Windows, i5/OS; DB2, MySQL, or Oracle; Apache, IIS, or lighttpd; we are commited to helping put PHP everywhere and anywhere.

Any questions, comments, feedback - feel free to drop me an email at andi at zend dot you know what.

PHP and Database Connection Pooling

During my recent vacation I worked on this blog entry which talks about connection pooling in PHP and some of the recent interesting results IBM published in a recent Zend Developer Zone article. I just saw Christopher Jones’ blog entry on a new feature coming in Oracle 11g so I’ve slightly adjusted it to include the news.

As is apparent from the likes of Yahoo!, Facebook and other large Web 2.0 companies who have billions of page views per month, PHP can indeed be deployed in a way which scales extremely well. Naturally with any technology that has to scale to huge volumes of traffic using large clusters of servers, there are always scalability issues that need resolving. Common issues that IT personnel deal with are monitoring and management of their clusters, application deployment, file system scalability, network topology, high availability and database scalability.

In my experience, databases have had a long history of being the typical bottleneck in PHP applications. There are many reasons for that including:
A) Dynamic Web applications tend to be heavily database driven and use the database as a searchable content store (CMS) and for transactional operations (e-commerce).
B) PHP developers prefer writing PHP code than SQL, thus often leading to sub optimal queries and moving the majority of the processing logic into PHP.
C) Web sites often serve millions of users, thus requiring PHP to deal with very heavy traffic. This in turn puts huge loads on the databases.
D) Lack of good cheap tools to find bottlenecks in database queries (such tools have always been very expensive. It will be interesting to see how MySQL's new service which also promises performance monitoring will change this).
D) PHP works best in multi-process environments such as Apache. There are many benefits to such a model especially stability, but it also typically requires each process to manage its own connection to the database.

What I'd like to talk about is the last point. Most production PHP deployments prefer to use persistent database connections in order to save the overhead of opening new connections on each request. This overhead doesn't only include the TCP/IP connection and the handshake with the database, but also server-side resource allocation such as processes, threads and the connection context which can take considerable time.

Using persistent database connections with the multi-process model creates a huge problem for large PHP clusters. Let's assume that the average PHP server is configured to spawn 200 Apache worker processes. For companies who have clusters with dozens or hundreds of PHP servers, this can end up being a huge strain on database servers who have to deal with thousands of concurrent connections. For example, a 100 server cluster will typically require 20000 concurrent connections. The performance hit on the server can be devastating for reasons including large memory consumption and crippling context switch overhead due to large quantities of threads and/or processes.

Such problems are not foreign to IBM's DB2 team. They have faced such multi-process architectures in the past and implemented a feature called connection concentrator. This feature efficiently handles large numbers of connections that have short-lived transactions which are very typical to PHP applications. Just recently IBM did some PHP specific benchmarks of this feature and wrote an article describing the results. These results really impressed me as they managed to easily scale to 10,000 concurrent DB connections (equal about to the size of a PHP cluster with 50 server) on a standard two-socket, dual-core processor Linux machine with 4 GB of memory. And not only was it on what is considered a lightweight machine for a database, there was memory to spare.

Although stability of the multi-process model has always been one of the main selling points for Apache, it has often been critiqued as not allowing for application-server like sharing of database connections between the various worker processes. This not only affects PHP but also other languages such as Python and Perl. Application server advocates have tended to point this out time and again, when in fact, it's a workaround for the database servers not scaling in this kind of scenario. Why should we be solving this problem by abandoning the rock-solid multi-process model rather than having the databases handle this common and increasingly growing runtime architecture? The traditional J2EE application server model is dying in favor of a more loosely coupled, robust and stable set of technologies that all interoperate and scale well together. Whether it's LAMP, OPAL, WIMP or other combinations of technologies, the world is moving away from monolithic systems to heterogeneous loosely-coupled systems. In such environments it becomes increasingly important for each part of the system to scale in various configurations and system architectures.

IBM has shown with this feature that they are successfully playing the loosely coupled game. Now Oracle are following and adding a feature which is also supposed to solve this problem and allow them to better scale in a heavy duty PHP environment. It will be interesting to see how their performance numbers stack up against IBM’s when the feature is released.

Zend/PHP Conference Keynote Speakers Announced

Today we announced the first batch of keynote speakers for the Zend/PHP conference. Not only is it great to see a large-scale PHP and Zend user such as Schematic on the list, but also having industry thought leaders such as r0ml and Chris Andersen volunteering their time to address the PHP community.

I'm particularly excited about Chris Andersen coming. The last time I saw him, Chris was still blogging about his upcoming book on The Long Tail Web site. Now the book is published, and having gotten my hands on a copy, I'm looking forward to getting it autographed at the conference!

As was the case last year, we believe we will be getting a good mix of both developers and business people at the conference, making it an interesting venue for all. Whether you're a long time PHP hacker or are a business looking for a very efficient way to build scalable modern Web applications, you will hopefully find the answers at our conference.

PHP 5 Power Programming passes 1 Million downloads!

When Stig, Derick and I decided to publish a PHP book, our editor proposed to do it as part of the Bruce Perens' Open Source Series. Not only is the series focused on open-source but in the true open-source spirit it is also made available as open-source under the Open Publication License. We loved the idea because it would give our work the widest possible exposure.
A few days ago I was informed that the PHP 5 Power Programming book has now been downloaded over 1 million times. Exposure no doubt. And considering that there are about 4.5 million developers out there, it's safe to say that probably between 1 in 5 PHP developers have actually laid their hands on it (or should I say their computer screens on it).
I'm glad we've managed to serve the larger PHP community by making the book so broadly available. As this book is targeting the experienced developer it's even more remarkable that so many people have downloaded it. The PHP developers community has truly matured.
You can find more information about the book on the Prentice-Hall web site.

CNET Article

I'd like to take this opportunity to respond to Edin's blog about what is stated in the CNET interview. As I mentioned in my response to Edin's blog entry, in the interview I never called out PHP 6 as a Zend product. I *did* talk about what's coming down the pipeline for PHP and mentioned that both Zend and Yahoo! have been contributing to the Unicode effort (which is true). One of my aims is to promote PHP proliferation and adoption, and talking about the support from organizations like Zend and Yahoo helps with that.

Zend is a commercial company working in the open source PHP community, as are Yahoo, IBM, Oracle and many others. There should not be an inherent conflict in that situation. We are very careful to acknowledge our relationship with the community – we point to the fact that there is such a large community of contributors to PHP as one of the key strengths of the technology (as can be seen in most of my presentations). We are also continually contributing significant Zend resources to the PHP community, working on PHP, Framework and the Eclipse IDE projects.

Our commercial activity is mostly around enabling larger corporations adopt PHP in a professional way, helping with architecture, training, development tools and runtime tools. The fact that these corporations adopt PHP further strengthens the community and creates opportunities for all of its participants.

Edin’s blog in part builds on a pre-existing perception that Zend claims more credit for PHP than it deserves. I acknowledge that there have been some unfortunate statements in the past that may have contributed to that perception. But that was years ago, and we have become very sensitive to that. Zend did not invent PHP, does not own PHP, etc. That goes for Zeev and I and all other Zenders. When we introduce ourselves we say that we are leading contributors to PHP, and I think we can make that case.

It’s also important to understand that it is *not* in Zend’s interest to claim that we are the only company behind PHP, that would make PHP look much smaller than it is which would be very counter productive to our quest for increasing PHP proliferation.

I think Stephen Shankland’s coverage of PHP and Zend are very positive, most of what he wrote is accurate. He may not be aware of the sensitivities in the community and so he describes PHP 6 in a way that it sounds like a Zend product. I did not say it, but Stephen writes the article from handwritten notes from the interview most probably a few hours later, so it is easy to see how that happens.

We will continue to promote PHP, the community, the contributors to it, the corporate support for it, and so forth. We are committed to make the whole community stronger, and will give continue to give credit to its many contributors.

PHP 5 Adoption has Tipped!

PHP 5 was released in July 2004. Since its release two years ago, there has been a lot of talk about its adoption rate and whether it will surpass the PHP 4 installed base. Even a very narrow Google query will point you to a large amount of articles and blog entries on the subject. In December 2005, I was interviewed for internetnews.com and already pointed out then, that judging by support calls we get for our various products at Zend, almost 50% of development in PHP is being done on PHP 5. Still many weren't sure whether to believe that statistic as Web crawling stats in favor of PHP 4 were still showing a completely different picture.

No doubt that it will still take some time for PHP 5 to reach PHP 4's existing installed base. The main reason for this are the numerous PHP 4-based applications that were built over the years, and if it's not broken, don't fix it. Even at Zend, we have some apps which don't need to be maintained anymore, so we still run PHP 4 for those. But pretty much all new development is done with PHP 5.

That said, I think when judging adoption, the most important statistic is still understanding what people developing and building Web sites with PHP are actually using today. Last week I decided to check up on our support statistics again and see how things have changed in the last few months. I was pretty amazed with the latest results. Although I knew many were waiting for PHP 5.1 before upgrading, its release in last November seems to have created a huge increase in PHP 5 adoption. In July over 74% of support tickets were by customers using PHP 5 (and remember we do support both versions). You can see the results (in %) in the following graph. PHP 4 and PHP 5 are of course inverses of each other.


I've added a trendline to the graph which gives a good idea of how adoption seems to be quickly accelerating since the release of PHP 5.1. I'm the last one to recommend trusting such extrapolations on data, but there definitely seems to be some kind of trend here, and I think it's past the point of no return.

P.S. - Thanks to Shahar Evron, from our support team and contributor to the Zend Framework for getting me these stats.

PHP represented at SIGMOD 2006

I was invited and spoke at the ACM SIGMOD conference in Chicago a couple of weeks ago. For those who aren't familiar with SIGMOD, it is ACM's Special Interest Group on Management of Data. The conference is probably the most important world-wide conference in this space where leading database researchers and developers meet to present their research papers.

It was therefore an honor and exciting to be given the opportunity of representing PHP as part of their industrial track. I gave a 25 minute presentation, focusing mainly on PHP's Web Services support and also talked about the XML Content Store work we're doing with IBM which is finding its way into the Zend Framework. The crowd was extremely well versed on XML and Web Services and I got some good questions.

As part of my track both BEA and Microsoft also presented some of their solutions. Erik Meijer from Microsoft presented a paper on LINQ which was very interesting. I've also posted my talk for those who are interested.

For lunch I caught up both with Hamid Pirahesh from IBM Research and with Erik Meijer. I had met Erik last about four years ago and I was surprised that I got somewhat the wrong impression of him back then. I had thought he was more of a purist language designer (e.g. C#/Java) but it was pretty cool to actually find out that he's a VB lover and that a lot of his thinking and the philosophy we followed when designing PHP are of similar nature. We both believe in ease-of-use and simplicity.

Hamid, Erik and I at lunch:



Now that dynamic languages are becoming first class citizens in the Enterprise, I believe we will be seeing more PHP coverage at such academic settings. Until next time...!

Eudora to Outlook Migration Update...

My previous blog entry regarding the Eudora to Outlook migration got some interest, so here are a few brief updates:

a) It's official. The MS Outlook import tool *really* sucks. Even if you first convert to Outlook Express, it just can't deal with a lot of the HTML emails and some customer Eudora tags. It left a lot of emails virtually unreadable. Thanks to an anonymous poster to my original blog entry, I found Aid4Mail which is commercial software that specializes in these imports. It worked MUCH better than the Outlook tools. It batch transferred all four years of email perfectly into Outlook. I just had to start it before going to sleep and in the morning it was done (remember this is over four years of email). Microsoft, you should consider licensing this technology! It's really in your benefit for such migrations to actually work.

b) As I mentioned in my previous posting, MS Outlook search sucks compared to Eudora. It was clear that I'd have to install one of the Desktop search products to get back some of the benefits of good search. I initially tried MSN Search as I thought it'd work best with MS Outlook. It was OK but it took too long to fire up after hitting enter on a search. So I installed Google Desktop Search and that works like a charm.

c) I mentioned that we use sanitizer at Zend which renames many attachments to attachment-xxxDEFANGED-doc. I hacked together a VBA script which went through all my years of email and renamed those attachments to be the correct original file name. I then also wrote a filter which would run on incoming messages and fix the attachment names. It's probably the worst code I've written but VBA is like PHP in that you don't have to know the language to use it :) You can find the filter at the following URL.
In order to use it, you should setup a Rule that filters incoming messages that have attachments. Also you need to create a directory c:\temp\attach which I use for temporary storage while extracting and reattaching attachments (Outlook doesn't support renaming attachments so you have to export and import them). Anyway, don't judge my coding skills by this code. It's pretty ugly and doesn't do any error checking. That said, it's been tested on a few hundred thousand messages and seems to be rock solid. It also logs what it does in C:\AttachmentFilter.txt. If you want the script that I used to bulk update all my emails, send me a note and I can send it to you. This code can be freely used for whatever purpose you wish.

So I think that completes my Outlook migration. Feel free to ping me if you have any questions.

Zend Framework 0.1.4 Released!

Today we published a new preview release of the Zend Framework.
This release comes after a somewhat long pause between releases which was due to many changes in the project. Everything is in place now including significantly better tools to manage the project including integrated bug tracker, milestone management, Wiki, etc... From now on we will be releasing early and releasing often. 0.1.5 with a new RewriteRouter is just around the corner and planning for the release after that has already started.
We've also gotten quite a few interesting proposals from the community as a result of proposal week and within a couple of weeks I believe these will be moving ahead in full steam.

I'd like to thank everyone who's helped get this out the door. The community did a great job helping out with this release including pitching in with the migration to our new project management tools. And of course thanks to the Zend Team for orchestrating the effort!

More details about this release can be found on the Zend Developer Zone.

Zend Framework invading Zend?

As many of you have seen a couple of months ago we launched a new and fresher look for the Zend Framework site.
When that went live, all the other Web site owners at Zend became jelous and wanted something just as nice. So it seems that slowly a lot of Zend's Web sites will start adopting this look&feel. The first in line was the Zend/PHP Conference Web site which just finished it's redesign and is a hell of a lot better than the 1990s look we had previously (no offense to its previous designers :) Anyway, I hear next in line is Zend Developer Zone, so stay tuned! We're finally going start getting rid of those nasty designs! :)

The End of an Era - Migration from Eudora to Outlook

Warning: This is a *very* long entry so skip if it doesn't sound like a subject that is of interest to you :)

When I first started using the Internet, Eudora was the thing. In addition to Trumpet Winsock, for getting Windows 3.1 online, Eudora was there side by side with Netscape Navigator, pioneering the way people communicated and shared information.

From those days onwards, Eudora has always been my preferred Windows-based email client and pine is what I use when I'm in a UNIX shell.
I had absolutely no problem shelling out the money for Eudora Pro because it was just far more powerful than any other mail client I had used.
I think due to the nature of my open-source involvement the amount of emails I've been getting over the years has always been ridiculous, so I would definitely consider myself an email power user. My needs probably don't match everyone's out there.


Some of the most important features to me in Eudora where:

a) Extremely powerful filters. In the early days it could filter by arbritary header (e.g. Delivered-To:) when Outlook and other mail readers couldn't.

b) Eudora folders that receive new email will popup and tile on the screen. This makes it easier to keep track of which one of the dozens of folders actually
has new email. In Outlook you don't really know if you have unread messages lingering which is often the case with me.

c) The search is probably the best part of Eudora. It is unbelievably convenient and even with the improvements in Outlook 2003, the usability and strength
of Eudora's search leaves it in the dust.

d) You can Ctrl-left-click on any subject and it will group all the emails in the selected thread for convenient reading. The same also works on people's emails
if you are looking for prior emails from the person who's email address you are clicking on. This is extremely convenient and doesn't require you to sort
all the emails in the mailbox.

So why did I end up leaving Eudora? Well, over the years I had all sorts of issues. The issues always made me keep a close look out for other solutions, and specifically Outlook which I had been used to using in parallel for years in a corporate environment.

What were some of the issues I encountered:

a) Attachments, attachments, attachments. This was probably my #1 problem. Eudora decided to keep all attachments outside the mailboxes. Initially this sounds great. Attachments are accessible through Windows Explorer and mailboxes are smaller and more compact. This caused huge problems though. All these mails were kept in the same directory and every few months, the directory became so huge that it was slowing things down significantly.
What I had to do was empty the attachments directory by moving all attachments to a backup directory. This caused problems because when I wanted to actually open an old attachment, I'd have to find it in Eudora, and then as it had moved, I would have to go and find it manually in one of the backup directories. With 5 years of emails and over 4 gigabytes of attachments this meant numerous backup directories and was just a huge PITA.

b) The Search was definitely a huge advantage, however I was always hoping for Google Desktop Search to support Eudora so that I'd finally enjoy blazing fast search.
Last year this wish almost came true, when Eudora OEMed a search engine and provided fast indexed search. The excitement soon turned into dissapointement because this search was constantly out of sync and I had to resync it manually from time to time. It became so annoying that at some point I just turned it off and went back to the older slower but more reliable search.

c) I started getting a lot of crashes in Eudora's internal viewer, mainly when viewing certain spam mail. I was pretty convinced that moving to the Word viewer would resolve this problem as Outlook uses the same engine. However, for some weird reason I just couldn't get it to work. It would always show a blank page. After opening a support ticket, I was told that starting over with a new Eudora.ini file should solve the problem. I tried it and it worked, but I just couldn't be bothered to reconfigure everything so I stuck to the internal viewer and became more careful with spam mail.

The actual migration wasn't too painful although it took me a few days until I was absolutely comfortable in making the move. I started with a pilot migration, did some testing of the filters and the imported messages (remember this is about 6 gig of email all in all), and when I felt as comfortable as I could get, I repeated the process as I had recorded it.

A few annoying things popped up during the migration process:

a) For some stupid reason MS Outlook isn't good at importing Eudora. It apparently looses the message dates, so the recommended way is to import to Outlook Express, and then important that into MS Outlook. Does that makes sense? Definitely not, but every place I found on the Internet, recommended that, so I didn't take any risks. Microsoft - Fix your MS Outlook import and and let the world know you did!

b) Apparently I was still on the "old" PST file format which blew up on me when I reached 2gig of capacity. It took me a while to fix it because it didn't allow me to erase messages (I guess there wasn't enough room to do the operation), and that was even without moving stuff to the Recycle bin. At the end I found some tiny messages which I could erase, and then quickly things started moving.

c) When I added the new kind of PST folder to Outlook, I erased the old one, and renamed the new one. For that point on I was constantly getting the error message "The operation failed. An object could not be found.". Well that's just great. Maybe if I'm not supposed to rename a folder then I shouldn't be allowed to do so via Folder->Properties... I finally found KB 312354 (http://support.microsoft.com/default.aspx?scid=kb;en-us;312354) which has instructions on how to solve the problen.

I've been on Outlook for a few days already and there are various pros and cons to the move.

Here are a few early thoughs:

a) I already mentioned earlier, that when new email comes in, there's no way of knowing which mailboxes have that new email (if you tend to have unread email in those mailboxes.)

b) We use procmail Sanitizer at Zend which renames attachments so that they can't just be opened (renames THISISAWORM.TXT.EXE to THISISAWORM.TXT.12345DEFANGED-EXE). This becomes extremely annoying to open (You need to save and rename the files). Eudora in one of its latest versions added some magic where it was actually possible to click on those links and the right app would launch. Unfortunately in Outlook that isn't the case. However, the cool thing about Outlook is VBA and the ability to write
scripts that manipulate your messages. I actually wrote a script that goes over all my 6 gig of messages and renamed the attachments to get rid of the DEFANGED portion of the name.
I will write about that in a separate blog entry as I believe it will be useful to many. It's probably the biggest reason why the move worked out to be the right thing to do.

c) Auto-complete of email address is by far better in Outlook than Eudora. In Eudora it would constantly screw up by saving invalid address which were mistyped at some point, but it seems that Outlook actually parses them and doesn't save invalid email addresses(e.g. "Andi - note the missing "). This led to a lot of frustration with Eudora.

d) I prefer to use an English locale on my machine and not Hebrew. Eudora couldn't show Hebrew due to this, and I do get Hebrew messages from time to time. To workaround the issues I'd open the message in the Web browser. In Outlook it just magically works.
e) While you can mark messages for follow-up in Eudora, it isn't quite as nice as the support in Outlook, where you can mark messages for follow-up, and then they appear in a separate view folder. Very convenient.

Well if you've gotten this far you are probably thinking of migrating too. I hope this info helps you. I felt it was important to blog about it because it was a big step for me and I thought that if others are looking into it, some of this info would help. I'm sure I'll discover more pros/cons in the coming weeks, and I'll definitely miss Eudora's search capabilities, but for now, especially after having been able to get rid of a few years worth of DEFANGED attachments, it does seem I made the right choice.

Eudora, I'll always remember you though!

Andi

Open-source Java?

At JavaOne Sun pronounced that it's planning on open-sourcing Java. So what does that really mean? Some might say that the Java source code has already been publicly available via Sun's source code license. But there is a big difference between "source code available" and "open-source". With OpenSolaris Sun did go down the right route not only by making the source code available but also with the open-source compatible license. This move has resulted in a bigger community around Solaris and even some 3rd party distributions such as Nexenta.

I've been asked by quite a few people whether I think open-sourcing Java is a good thing for Java. It's definitely a tough question and I don't think there's one good answer. That said I did collect some random thoughts I have on the subject.

I think one of the biggest advantages to making this move would be to significantly grow the developer's community around Java, and by doing so, increasing the manpower contributing to its maintenance. The JDK currently has thousands of open bug reports, and that is only counting the core. One thing my PHP experience has taught me is that one of the things the open-source communities are really good at are with squishing bugs. I wouldn't be surprised that after Java goes open-source, that those thousands of open bug reports are reduced to just a few hundred in a very short time period. Some might suggest that the source code has always been available but the truth is that such communities won't form unless the license and the project actually are designed to officially accept contributions and support an open-source community.

Another interesting outcome would be the ability to reuse some of the source code in non-Java related projects. There is some good code in Java and I think some of it could be reused in other projects (depending of course on license compatibility). As some of the smartest people worked on this code I think this could be of value to the broader open-source community.

Last but not least, I think that the availability of the source code could lead to innovation happening around it. This could be both in form of performance and functionality, but I think what would be especially interesting are domain specific versions of Java. I think it's quite foreseeable that you'd see tailored Java versions which are very specific to certain applications and industries. I'm already looking forward to seeing some interesting domain-specific offsprings.

So definitely lots of advantages but I also have some concerns around such a move. If the source code would be repeatedly forked (which as I previously stated could also lead to some interesting domain specific implementations), then incompatibilities between various Java implementations might make it extremely hard for ISVs to create supportable apps for these platforms. You'd have a similar situation (but even harder to solve) as we have in the Linux world where most ISVs officially support Redhat&Suse, but have to unofficially test on at least another half-dozen of distributions such as Fedora, Ubuntu, Debian, and others. This kind of fragmentation in the Java space and the wide distribution of incompatible Java versions could lead to a big headache for Java vendors and customers.
Another concern would also be backwards compatiblity. Up to now, Sun has done a pretty good job on backwards compatibility in Java. Having more of an open process might make keeping that balance harder. We have seen similar things happen in the PHP space, and it hasn't been easy to keep the balance between backwards compatibility and fixing issues.

Overall I do think it's a good move and the right move. I believe Sun can use open-sourcing Java as tail wind for a lot of things it's doing well such as Hardware & Solaris.

Those are some of my random thoughts on the subject. No doubt they are incomplete. I'd be happy to get comments pointing out other pros/cons which I haven't thought of, or reasons for why my thoughts are wrong :)

And I can't end this Java-centric blog article without saying Happy PHP'ing :)

Zend Framework surpassed the .NET Framework!

Mike brought to my attention that according to Microsoft sources the Zend Framework has surpassed the .NET Framework. Think it's a joke? Go to MSN.com, search for "framework", and you'll see the Zend Framework appears first, and .NET Framework second.
And we're not even close to a release yet! :)

New Zend Framework Website look launched (almost)

After my talk at Scale 4x, the Southern California Linux Expo, Roy Rubin from Varien got in touch with me. A synposis of what he said was "The Zend Framework is very exciting, but its Web site is not". He was being polite as I'm sure he really want to say "Your Web site's look sucks".
In the spirit of community, Roy offered to contribute to the Zend Framework by having his team help with a new design. From seeing the sketches it was clear that Varien could deliver something far more appealing (a.k.a sexy) than what we had created. So after hard work by them and Mike from the framework team, we've almost got it done.
In the spirit of release early, release often, we've decided to launch it a tad bit prior to its completion. We still have some work to do in order to get the manual integrated into the new site. The new manual look is MUCH nicer. So sorry, it looks a bit weird, but actually it's kind of cool in that it allows you to jump between the old and new look and actually see the huge difference in what the Varien team did.
Another thing we REALLY wanted to have done for the launch was a bug tracker. It's still top priority and we'll hopefully be able to launch that in the coming days (using the new look of course).
So I'd like to thank everyone who helped on this project especially Ben Blumenfeld and his team who did the design. It shows how outside contributions can make a big difference.

PHP coming to i5/OS

As many might have seen, Zend and IBM announced the support for PHP on i5/OS, or previously known as AS/400. Although this platform might be foreign to many in the PHP community, over the past few years I have discovered quite a few who have either worked with PHP on this platform by following IBM's Redbook instructions or have installed PHP on a UNIX/Linux server and then interoperated with the i5/OS.

As part of this project we will be providing all i5/OS users Zend Core for i5/OS for free. This will very much lower the barrier of entry to get up and running with PHP. We will also be providing ways for them to access their existing code-base by creating language bridges (such as an RPG bridge), native drivers for accessing DB2/400, and various other bridges to operating system services. I believe this initiative will have a huge impact on the i5/OS community, which has primarly been offered Java as a way to Web and Web Service enable their platform, but as most of us PHPers know, Java is just too damn hard and takes too much time... :)

There are many interesting parallels between the i5/OS developers community (mostly developing in RPG) and the PHP community. First of all it's also a very passionate community which deeply cares about their platform. Having been at their last user group conference it's clear that it translates into a very vocal and honest dialogue. Second, it is a community that also tends to have less formal computer science education, but learns what it needs in order "to get the job done". So I look especially forward to seeing the i5/OS and PHP communities mix on the basis of these and other similarities.

I've already talked to some of you in regards to the PHP & i5/OS work you've been doing. I'd love to hear from additional people who might want to use PHP on this platform.

The early access version (no language bridges yet) can be download from the Zend Web site.
You can catch me at andi@zend dot youknowhwat.

ZEND/PHP CONFERENCE & EXPO 2006 - CALL FOR PAPERS

I'm happy to inform that Zend is holding its 2nd annual Zend/PHP conference in October 2006 in San Jose, California. Last year was a great success which will surely be hard to beat, but I hope that this year will be at least as good.

More information for people who are interested in participating and speaking at the conference can be found here. Submissions need to be made before June 15th, 2006.

See you there!

Quick Zend Framework update.

Just wanted to give a quick update on Zend Framework.

We've already received a lot of great feedback. Thanks everyone!
We have a few problems in this initial release, some due to last minute changes (like Zend_Filter refactoring) and other things that slipped by or were planned for after the first code drop. So let me just give a quick recap of what those problems are so that you know what you can expect to be fixed in the coming days:

- Fix documentation (we mistakenly included old docs for ZDb_DataObject which doesn't exist, and didn't bundle the Controller docs). Thanks to the power of community there are some underground docs online now :) http://www.akrabat.com/2006/03/04/zend-framework-front-controller/ but we will also bundle the official docs in the next drop. We are also planning on beautifying and integrating the docs more closely with the Web site.

- Resolve issues that crepped in due to last minute Zend_Filter refactoring. Thanks to Chris Thompson who sent in instructions on getting it to run again:
1. In the file "Zend/Http.php" change the line
"$allow = Zend_InputFilter::HOST_ALLOW_DNS | Zend_InputFilter::HOST_ALLOW_LOCAL;"
to
"$allow = Zend_Filter::HOST_ALLOW_DNS | Zend_Filter::HOST_ALLOW_LOCAL;"
2. In the file "Zend/Filter.php", at the top add the line "include_once 'Zend/Filter/Exception.php';".
3. Also in the file "Zend/Filter.php", comment the line "throw new Zend_Filter_Exception('Internal error: IP determination failed');" (did not look into why -- I just commented it out)

- Setup NEWS file so that people can track the changes/fixes between code drops.

Some general issues which we're planning to resolve in the coming days:
- Bundle framework unit tests with the next drop
- Setup mailing list archives
- Provide anonymous SVN access so that people can pull current versions by themselves
- Fix Web site link to CLA
- Add instructions on how to subscribe to the mailing list in a prominent place on the framework site.

Thanks all for the feedback. It's pretty amazing how many people actually downloaded and looked at this stuff on a weekend. Late Friday evening is a developer friendly release :)

PHP Collaboration Project reaches major milestone

Last October, we announced the PHP Collaboration Project. The idea was to collaborate around selected areas in PHP to further enhance the overall PHP eco-system. At that time we mentioned three projects which we were planning to start. The Zend Framework, an Eclipse-based PHP IDE framework, and best practices. I'm happy to say that today all these three projects have made significant progress and with that we have reached the first milestone in this initiative.

The past few months have been great. We had many more than expected individuals and companies who showed interest in contributing. We managed to gather a relatively small but solid team of people to move all these projects forward. I'd like to thank everyone who has helped and worked hard (often on their free time) to make this happen.

We have finally updated the static PHP Collaboration Project page which now points to the individual projects which we have started:

- The Zend Framework - We have released a preview release of the Zend Framework. I'd like to thank all the initial contributors who have been of tremendous help in getting this off the ground. There is a lot of work still to do, but after having already seen four applications build with the framework, it is clear that it already includes some very cool and useful modules.

- The Eclipse PHP IDE framework - We have submitted a proposal to the Eclipse Foundation for a community based open-source PHP IDE framework. If all goes well in next week's creation review, we will be submitting the initial code drop shortly thereafter. In the spirit of Eclipse, we have already reached out to community members, including the PHP/Eclipse project, and have received great interest to join the effort.

- The Zend Developer Zone - We have launched our new developer zone. The goal of this developer zone is to provide best practices with high-quality content from partners and the community. We already have many companies and invidiuals lined up to collaborate around this site who will share their knowledge for the benefit of the PHP community. The site will cover many aspects of best practices including Zend Framework use, interoperating with other technologies, and general PHP best practices around subjects such as security. The Zend Developer Zone is still under active development so expect further changes in the coming week.

I'd like to thank again the many people who have helped out with these three projects. I have no doubt that they will benefit the PHP community.

Happy PHPing!

Going to mashup camp...

Monday and Tuesday is Mashup camp. It's what I call organized chaos. Initially a Wiki was put up, a few emails were sent around, and very quickly people stepped up to participate, contribute and sponsor the event.
It's still not completely clear what will be going down there but from scanning the list of attendees I do think that it'll be a noteworthy event where us geeks (and some impostors) will be trading secrets and ideas on how to take the idea of mashups and Web Services to the next level. It's becoming increasingly clear that mashups can have real business value (and OK, some are just cool), and the skies the limit when it comes to mixing and matching widely exposed services and data.
Zend and Yahoo! Developer Network are co-hosting a reception with mashup food (take a wild guess what that means). So please stop by!
We'll also be showing some of the functionality of the upcoming Zend Framework especially the components which are specifically targetted at consuming Web Services and will showcase how the simplicity and power of PHP is a perfect platform for mashups. As part of that we will showcase an Application Wiki written by IBM using PHP and some Zend Framework pieces. It very much shows the Enterprise mashup concept i.e. mashups can solve real business problems.

See you there!

Expose for Firefox...

This might be old news, but I was just reading Stephen O'Grady's blog and found mention of a very cool Firefox extension called Reveal.
Reveal is to Firefox what Expose is to Mac OS X (don't kill me if that didn't come out well :)

I highly recommend it. It's making it much easier to navigate my millions of Firefox tabs.

php|architect's Pro PHP Podcast

Today Marcus Whitney hosted me on his Pro PHP Podcast show. I was really impressed by his professionalism and think he had some very good questions. I'd like to apologize to the people who were trying to listen in on the live session which was cancelled after I got cut off twice. There must have been some problem with the "conference" mode of the Podcast. When I redid the interview with Marcus directly, it all worked fine.
For those who are interested, you can listen to the interview here.
If you have any further questions or comments, feel free to email me at andi@ you know where...

I belong to the ZCE 1000

Not to be confused with the Fortune 1000 :)
It's been about a year and a half, since we launched our Zend Certification program. Within a few days we'll be surpassing the 1000 mark of certified engineers. To celebrate the event, we have created a new logo, themed as ZCE 1000, for use by the first 1000 certified engineers. If you're past the 1000 mark, we'll sue you if you use it :) Just kidding...
Here's what it looks like

Woody Allen's Best!

Just got back from seeing Woody Allen's latest movie, Match Point. Although I've enjoyed some of his movies, I've never been one of his greatest fans. This movie though is truly a masterpiece. It's completely different from his other movies. It's not in New York but in London, not a comedy but a drama, and just in general, is quite different from any of his other movies.
I highly recommend you see it! It's one of the best movies I've seen in the past few years.

PHP is a great language!

One thing I have noticed, is that quite often, PHP developers who are seeking for new PHP features are prematurely trying to implement them in C. Although, there are definitely cases where you want to write your code in C, I think in some cases PHP is too quickly dismissed. Developing features in PHP not only takes less time, but is also less prone to bugs, easier to maintain, and more stable and secure. Also, in the majority of cases, whether such a feature were implemented in C or PHP, would not make a significant different to overall application performance.

It is for this reason, and for reasons of ease-of-deployment, that in the Zend Framework we are actually taking full advantage of the PHP language. We want to showcase what the language can do, and not drop into C when we feel challenged. Also, the fact that we are implementing everything in PHP will make it also much easier to deploy, as we will aim not to have dependencies on non-standard extensions. Of course we will depend on what we consider mainstream PHP 5 extensions such as SimpleXML, DOM, PCRE, sessions etc.

ZSearch, one the components we presented in the php|architect Webcast, is a great example of this. Despite the fact that a Lucene implemention does a lot of bits and bytes file access, we implemented it fully in PHP. Is it as fast as a C implementation? Most definitely not. Is it good enough for use on high-traffic Web sites. We believe so. Same can be said for our PDF implementation, ZPdf. As we wanted to make it eas easy as possible to manipulate Pdfs, without requiring users to have to recompile PHP, we decided to implement it 100% in PHP. Not only are we implementing the creation of Pdfs, but we are also going to enable reading Pdfs, traversing them, manipulating and then writing the changed document back to disk. The implementation will even include a PHP-based LZW implementation needed to support the latest Pdf standard.

I believe PHP is a great language and from my experience a lot of needed functionality can be written and maintained much better on the PHP level, and except for limited examples, we all know that most often the bottlenecks in Web apps don't lie in a bit more or a bit less code, but in a lot of variants of the stack such as database , file system and network acccess, and more...

Long live PHP!