If you haven't heard of the archive team I suggest you look them up and even become an archiving activist or even hero!
I had heard of the archive team before and like many, I believed that this rogue group are like over "energetic hackers" with an agenda to preserve the lives of people that companies destroy.
This is an agenda I fully support but had not been inspired to take part until recently when I had the absolute pleasure of listening to a keynote speech by Jason Scott, one of the founding members of the archive team. Jason sums up himself as an "energetic hacker" and he is a man who shares many of my beliefs which I sum up in this post.
1) Publishing is moving too fast for archives to keep up.
Jason compares archiving peoples lives on the web in systems like geocities, MobileMe and others like trying to catch fireflies. They fly up and while you have time to say "How Pretty", you have no time to realise what the firefly does and its value to the community.
Once you do realise the value, they are already gone.
This is true of nature, buildings and even people themselves! In many cases there is nothing we can do, but what about digital information?
Cathy Marshall presents a survey carried out at the Library of Congress of digital lives with some children and one quote stuck out:
"Why don't we just save facebook as this is our diary, yearbook, guinness book of records..."
This is absolutely something that can be done, and something that the archive team are doing...great... but why aren't others?
One word... Policy
2) Just get on with it, ask for forgiveness later.
It is much easy to ask for forgiveness that it is to reconstruct a building that has been knocked down. Once it is gone, nothing can be done. Something that has been saved, can still be removed...
Jason and archive team apply this policy, archiving whole websites and peoples public content, indexing it and making it available on torrent sites. And this has saved a lot of peoples content from the trash can.
Again, POLICY is a blocker to people just doing this, the other blocker is the belief that people will not want you to archive and share their data... let's address that...
3) People know what they want, so ask them!
Question for you to consider: Should your medical record be shared?
What if it was shared with researchers worldwide so that a cure might be found quicker?
What if your medical record became the top hit for your name of google?
It is this last point that seems to be key to most people, many people don't mind historical information being made available, however they just don't want such information to be any more prominent that it was.
Finally the hardest thing to archive is to allow the user to still be in control of their data and allow it to be deleted if they want it too. A big problem with all of these systems is that companies and archives assume ownership of peoples data, it is not their data, it belongs to the curators of the data and control should remain with those people.
If you lend a dinner set to a friend or neighbour for a big party they are planning, you expect to be able to get it back, not for them to take control of it and then throw it away instead of give it back to you.
POLICY...
It is time that policy was changed. It definitely should not get in the way of progress!
Thursday, 5 July 2012
Wednesday, 22 February 2012
Drupal 7 - From Blank to Working
I have never set up a site from scratch in Drupal before, but I am impressed how easy it is to use for users who don't want to delve into a terminal and try and understand templating that way. Problem is getting from a fresh install to basic site is actually quiet hard! This blog lists the steps I went through to make it work, it already assumes you have set up an admin account. The following steps look at building a simple site in the Bartik theme.
1) Set up your development environment - Make sure you have two browsers open, one in Incognito/Private browsing mode such that you can see changes to users who are not logged in.
2) Login and create a main page, this is a basic page and will be called something like node/1. Via Configuration -> Site Information you will need to set it as a home page (set the other options while you are here).
3) Change the theme to Bartik in Appearance and then click settings to customise your colour scheme and default icon (note that icons need to be the correct size).
4) Go to People and then click the Permissions tab. Make sure that all users can search your site. Your can then add the search block (via Structure -> Blocks) to the header (or other) section of your page.
5) Go to Structure -> Menus and click "list links" against "user menu". Then click add link and add a Login link with the url user/login and enable this. You can do the same with the url user/register to get a registration link. Refreshing the non-logged in page should now show these links at the top right. Turning off registration can be done via Configuration -> Account Settings.
6) From Structure -> Blocks create a new block with the following content and add it to the header section of your page:
This should set you up a basic and usable site.
1) Set up your development environment - Make sure you have two browsers open, one in Incognito/Private browsing mode such that you can see changes to users who are not logged in.
2) Login and create a main page, this is a basic page and will be called something like node/1. Via Configuration -> Site Information you will need to set it as a home page (set the other options while you are here).
3) Change the theme to Bartik in Appearance and then click settings to customise your colour scheme and default icon (note that icons need to be the correct size).
4) Go to People and then click the Permissions tab. Make sure that all users can search your site. Your can then add the search block (via Structure -> Blocks) to the header (or other) section of your page.
5) Go to Structure -> Menus and click "list links" against "user menu". Then click add link and add a Login link with the url user/login and enable this. You can do the same with the url user/register to get a registration link. Refreshing the non-logged in page should now show these links at the top right. Turning off registration can be done via Configuration -> Account Settings.
6) From Structure -> Blocks create a new block with the following content and add it to the header section of your page:
<style type="text/css">
.breadcrumb {
display: none;
}
</style>
This should set you up a basic and usable site.
Friday, 20 January 2012
Making Debian Changelogs from Github repositories
One of the many things that irks me is the gap between good developers who put all their code on platforms such as GitHub, and those who then actually bother to put some effort into packaging up their code for easy platform installation.
I have come to the realisation that this is mainly due to the pedantic nature of packaging formats and platform lock in. One such example is the exacting format of the debian changelog...
GitHib2Changelog is a bit of code that I knocked together to help in this situation. It takes a GitHub repository URL and builds a debian changelog from the repository commits and tags.
By looking at the tags and commits it works out which commits are related to which tags (something GitHub APIv3 doesn't do) and then outputs this directly to you already formatted.
The service is built in php, and is web based with both a pretty front end and API access.
Ironically, since i've now committed the code to GitHub here I now need to use the service on itself and build the easy to install packages. More on that soon...
I have come to the realisation that this is mainly due to the pedantic nature of packaging formats and platform lock in. One such example is the exacting format of the debian changelog...
GitHib2Changelog is a bit of code that I knocked together to help in this situation. It takes a GitHub repository URL and builds a debian changelog from the repository commits and tags.
By looking at the tags and commits it works out which commits are related to which tags (something GitHub APIv3 doesn't do) and then outputs this directly to you already formatted.
The service is built in php, and is web based with both a pretty front end and API access.
Ironically, since i've now committed the code to GitHub here I now need to use the service on itself and build the easy to install packages. More on that soon...
Thursday, 19 January 2012
DepositMOre - The Prototype
Building on the success of DepositMO and SWORDv2, I thought it would be a good idea to put a quick HTML5 client together to save myself some pain.
The basic premise of this web-based client is to automatically search for "your stuff" in a number of ways and then allow it all to be submitted to a repository in one click.
First target for me was www.easychair.org. This service is used as an online conference submission and review system. In a nut-shell if an author wants to get accepted into a conference, easychair is one system which they WILL have to battle with in order to submit their content. As a result there is a strong potential that easychair knows about many publications which should also be present in other systems.
From the main screen in easychair it is possible to navigate and find the many conference publications which you have submitted. Each publication is tied to a conference and it can take a substantial number of clicks to navigate between each publication.
DepositMOre is a modular system which is intended to be a home for many services which locate your publications. The first module to be developed is for easychair.
By simply providing your login credentials to the DepositMOre system, it will not only list all your authored items from easychair but also check if these are present in your locally detected repository. If they are not deposited, and they should be then one click will do this for you.
A combination of HTML5 and SWORD2 make this process quick and seemless! Multiple items can be submitted at once and as each are submitted you can instantly click a link to your item and can view it in the repository.
The following video gives a demo of the prototype in action. We hope to continue development with the support of a funded project.
The basic premise of this web-based client is to automatically search for "your stuff" in a number of ways and then allow it all to be submitted to a repository in one click.
First target for me was www.easychair.org. This service is used as an online conference submission and review system. In a nut-shell if an author wants to get accepted into a conference, easychair is one system which they WILL have to battle with in order to submit their content. As a result there is a strong potential that easychair knows about many publications which should also be present in other systems.
From the main screen in easychair it is possible to navigate and find the many conference publications which you have submitted. Each publication is tied to a conference and it can take a substantial number of clicks to navigate between each publication.DepositMOre
DepositMOre is a modular system which is intended to be a home for many services which locate your publications. The first module to be developed is for easychair.By simply providing your login credentials to the DepositMOre system, it will not only list all your authored items from easychair but also check if these are present in your locally detected repository. If they are not deposited, and they should be then one click will do this for you.
A combination of HTML5 and SWORD2 make this process quick and seemless! Multiple items can be submitted at once and as each are submitted you can instantly click a link to your item and can view it in the repository.
The following video gives a demo of the prototype in action. We hope to continue development with the support of a funded project.
Technologies Used
- HTML/Javascript/JQuery/PHP
- SWORD2 PHP Library - Stuart Lewis - https://github.com/stuartlewis/swordappv2-php-library/
Monday, 31 October 2011
A little preservation watch tool for DROID users
Ever wondered what has changed in each new signature file released for The National Archives DROID tool?
Want a way to find out what objects a new signature file might affect or reclassify?
I have collected together all the available DROID signature files (still want more) and produced a little preservation watch tool that surmises changes between signature file versions. A summary is produced which outlines the signatures and file formats added to each new signature file. Additionally by selecting any two of the signature files, a user is also able to compare two specific versions.
Soon to be added will be the ability to subscribe to an email or RSS feed which alerts of new signature files and changes, allowing active preservation watch.
In order to tailor this more to a users n eeds I'm contemplating allowing selection of specific extensions/formats which users care a lot about and producing an alerting service which focusses on changes to only these types.
Thoughts welcome...
Monday, 18 July 2011
More on File Identification Tools
Since I last wrote a post on this back in March I have started some work for the Open Planets Foundation. As I said in my previous post, I see no reason to have too many unmaintainable tools when we could just pick the best one... the problem is making this choice (for some).
Which tool?
Simple - The one which is currently the most widely adopted... file.
Opinions may vary on this however ALL of these arguments talk about the feature set of a particular tool or the slowness of the tool when scanning billions of files.
Feature Sets and Ease of Use
File is a very simple tool which offers a mime-type and limited metadata exposing of the file types about which it knows. It only accepts single file execution, however you can wildcard it's input in the linux shell and it executes extremely quickly. In my testing file took 2.375s to identify 1000 files, that's 421 files a second (see the comparison to DROID and FIDO @ http://www.openplanetsfoundation.org/blogs/2010-11-03-fido-%E2%80%93-high-performance-format-identifier-digital-objects#comment-75).
Other tools offer more power in other ways, so DROID fits well with The National Archives (UK) Digital Continuity project, providing a PRONOM identifier and mapping back to tools which can perform many operations on these files.
DROID is an ever improving tool as the underlying data (number of files it can identify) expands. However certain decisions have led to it becoming difficult in recent times to simply profile a single file. Rather it has become a tool which is now quiet heavily integrated with other systems rather than loosely coupled.
FIDO resulted from this realisation that DROID was becoming too slow and painful to profile a single file. Originally written in python, FIDO performs the same classification as DROID (using the same signatures) in a much shorter time and provides the PRONOM PUID as output. FIDO provided a great proof of concept that this operation could be quick however suffers from the problem that someone has wrapped the current release (0.9.5) of FIDO in java. This slows it down significantly due to having to launch the JAVA VM!
FITS - The file identification tool set.
This final tool pretty much wraps everything, some really useful and detailed output can be gained as a result of running not only all the identification tools, but also classification tools like exiftool and ffmpeg. Lots of detail, really slow! Also, more so than any of the others, FITS suffers from the problem of being up to date more than the others.
The problem with FITS is that it wraps the other tools to provide one output. In the case of FITS they still wrap DROID v4 with a very old signature file. They chose DROID v4 as this was the last version with decent command line execution, which is (i'm guessing) the way they call it. FITS wraps a great number of other tools in it's distribution as well, such as the exiftool, which already have package managed versions, thus all the tools FITS uses are being constantly dated by new versions which then required the effort to wrap them. FITS is a great attempt at a very valuable tool, however the problems with the tools it bundles being updated constantly is likely to cause many maintainability problems.
Along with all the other tools, FITS suffers from the fact that it doesn't update the DROID signature file (a format which hasn't changed) automatically. This is a simple way for a tool to keep up to date, we should not have to rely on the users doing this when the tool should just be doing it for you! People are lazy these days and expect the package to just work, or their to be an available update of the whole package (akin to App Store approaches). The users are right in this respect BTW!
File, the tool I haven't mentioned for a while is packaged managed by every widely used platform now, so if there is an update, people are alerted to it. From this point it only takes one click to download the latest, greatest and fastest version.
Packaging
Like Weird Al's song "Albuquerque" I have finally got to the point (but not the conclusion) ... Packaging.
In order to keep users happy we MUST learn to allow them to download and use tools which suit them.
Personally I'm fed up of JAVA integration, when to install a package you have to first install Maven, then install something else, then do this..... blah blah bored....
We need to start packaging cross platform tools inside one click install MSIs (windows), RPM (redhat) and DEB packages (Ubuntu/Debian and the rest of Linux).
I don't care if these packages install dependencies but the user shouldn't have to take more than one step to install a tool.
Futher the tool should either self update, or the user should be prompted to update it via the package managing option which their operating system already contains.
Using packages, FITS could simply be a very small meta package which depends on other packages, thus keeping the whole suit of tools up to date... independently.
Conclusion
* Yes a tool should be fast
* Yes a tool should be feature rich
* Most of all, you should be able to install it and keep it up to date easily!
What's Next
The billion dollar question, for me I've done some performance testing of the various tools and decided that speed is due to features. As a package gets bloated and feature rich, it becomes slower! The faster it is the simpler it is.
What i'd like to be is to make the fastest one (file) feature rich without bloating it and slowing it down. Also file is already package managed which saves me what appears (according to the other tools) to be a very hard job.
Some investigation on this aim is likely to follow, along with some requirements gathering and classification on critical features before things move forward.
Monday, 14 March 2011
Preservation Tools - Moving Forward
Over the last number of years, JISC and other bodies have funded a number of digital preservation projects which have resulted in some really valuable contributions to the area... now is the time to realise the benefits of this work and provide a digital preservation experience to everyday users.
To achieve this a not insignificant amount of work needs to be undertaken, namely to identify key applications and separate these from the complex systems into which they have been built. Alternatively many applications now need re-thinking and the best bits built into system which have super-ceded these applications.
File Format Identification Tools
File format identification now has a number of tools available, each with their own advantages and disadvantages, in no particular order they are:
DROID:
- Started out as a tool to identify file types and versions of those types. :)
- Each file version was assigned an identifier which could be referenced and re-used. :)
- Identification of file was done via "signature", not extension matching. :)
- Became complex as it was adjusted to suit workflows and provide much more complex information which few people understand or want :(
- Added complexity increased the time required for each file classification, no longer a simple tool :(
FIDO:
- A new cut down client which takes the DROID signature files and does the simple stuff again :)
FILE:
- A built in Unix tool installed on every Unix based system in the world already! :)
- Does not do version type identification :(
- Does not provide a mime-type URI :(
- Very quick to run :)
- Has the capacity to add version type identification and there is a TODO in the code for it! :)
With the PRONOM registry now looking at providing URIs for file versions, why can't we stop coding new tools and change the FILE library. This way it could handle the version information and feed back the URIs if people want them. I've looked briefly into this and the PRONOM signatures should be easy to transport and use with the file tool.
If I get time I might well have a go at this and feed it back to the community.
Friday, 4 March 2011
Installing Kinect on Ubuntu (A full guide)
1) sudo apt-get install libglut3-dev build-essential libusb-1.0-0-dev git-core
2) mkdir ~/kinect && cd ~/kinect
3) git clone https://github.com/OpenNI/OpenNI.git
4) cd OpenNI/Platform/Linux-x86/Build
5) make && sudo make install
6) cd ~/kinect/
7) git clone https://github.com/boilerbots/Sensor.git
8) cd Sensor
9) git checkout kinect
10) cd Platform/Linux-x86/Build
11) make && sudo make install
12) go to this page at openNI to download the latest NITE release for your platform:NITE download page or for the impatient:
32-bit
64-bit
13) Save the NITE tarball to ~/kinect and untar it
14) cd ~/kinect/NITE/Nite-1.3.0.17/Data
15) Open Sample-User.xml,Sample-Scene.xml and Sample-Tracking.xml and replace the existing License line with the line below: NOTE: this is case sensitive!
< vendor="PrimeSense" key="0KOIk2JeIBYClPWVnMoRKn5cdY4=">
16) Repear step 15 and replace the existing MapOutputMode line with the line below in all 3 files.
< xres="640" yres="480" fps="30">
19) sudo niLicense PrimeSense 0KOIk2JeIBYClPWVnMoRKn5cdY4=
20) cd ~/kinect/NITE/Nite-1.3.0.17/
21) sudo ./install.bash
22) make && sudo make install
23) cd ~/kinect/NITE/Nite-1.3.0.17/Samples/Bin
24) sudo adduser YOURNAME video
25) nano /usr/etc/primesense/XnVHandGenerator/Nite.ini by uncommenting the two config parameters it contains
26) sudo nano /etc/udev/rules.d/51-kinect.rules
# ATTR{product}=="Xbox NUI Motor"
SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02b0", MODE="0666"
# ATTR{product}=="Xbox NUI Audio"
SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02ad", MODE="0666"
# ATTR{product}=="Xbox NUI Camera"
SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02ae", MODE="0666"
27) sudo /etc/init.d/udev restart
28) cd ~kinect/Nite-1.3.0.17/Samples/Bin/
29) ./Sample-PointViewer and PLAY
2) mkdir ~/kinect && cd ~/kinect
3) git clone https://github.com/OpenNI/OpenNI.git
4) cd OpenNI/Platform/Linux-x86/Build
5) make && sudo make install
6) cd ~/kinect/
7) git clone https://github.com/boilerbots/Sensor.git
8) cd Sensor
9) git checkout kinect
10) cd Platform/Linux-x86/Build
11) make && sudo make install
12) go to this page at openNI to download the latest NITE release for your platform:NITE download page or for the impatient:
32-bit
64-bit
13) Save the NITE tarball to ~/kinect and untar it
14) cd ~/kinect/NITE/Nite-1.3.0.17/Data
15) Open Sample-User.xml,Sample-Scene.xml and Sample-Tracking.xml and replace the existing License line with the line below: NOTE: this is case sensitive!
< vendor="PrimeSense" key="0KOIk2JeIBYClPWVnMoRKn5cdY4=">
16) Repear step 15 and replace the existing MapOutputMode line with the line below in all 3 files.
< xres="640" yres="480" fps="30">
19) sudo niLicense PrimeSense 0KOIk2JeIBYClPWVnMoRKn5cdY4=
20) cd ~/kinect/NITE/Nite-1.3.0.17/
21) sudo ./install.bash
22) make && sudo make install
23) cd ~/kinect/NITE/Nite-1.3.0.17/Samples/Bin
24) sudo adduser YOURNAME video
25) nano /usr/etc/primesense/XnVHandGenerator/Nite.ini by uncommenting the two config parameters it contains
26) sudo nano /etc/udev/rules.d/51-kinect.rules
# ATTR{product}=="Xbox NUI Motor"
SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02b0", MODE="0666"
# ATTR{product}=="Xbox NUI Audio"
SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02ad", MODE="0666"
# ATTR{product}=="Xbox NUI Camera"
SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02ae", MODE="0666"
27) sudo /etc/init.d/udev restart
28) cd ~kinect/Nite-1.3.0.17/Samples/Bin/
29) ./Sample-PointViewer and PLAY
Monday, 22 November 2010
Hot Topics in Scholarly Systems
Since I last wrote a blog post the world has been going through some harsh times where cutbacks and simplifications have been essential. The phrase "Throw money at it" no longer applies to anything and all of a sudden organisations as well as people seem far more keen to share than before (although we are still not fully open and sharing, mostly it's organisations wanting stuff without sharing themselves, but we'll get there).
Anyway enough of that, what is actually happening?
Well I am very proud to be at the forefront of an international effort to hold a series of scholarly technology meetings focussed on solving institutional problems. These meetings, known as the Scholarly Information Technical Summit (SITS) meetings, are being held in alongside many international conferences over the next 2 years and are being backed by all the major international funding bodies. See http://bit.ly/Scholarly_Infrastructure_Technical_Summit for more info.
There have now been 2 meetings, although SITS only came about because the first one was so successful. Each meeting conforms to the Open Agenda (see wikipedia) principal and is chaired likewise. This leads to the agenda being very pertinent to the people in the room and often creates conversation critical to the forward momentum of some of the technologies discussed. In the next few paragraphs I'm going to try and summerise the hot topics from the first meeting:
SWORD - Put stuff in a repository
SWORD has undoubtedly been a huge success, it's simple and well supported by many publishers and publishing software (including most notably the Microsoft office suite via the author add-in tool http://research.microsoft.com/authoring). There are however some problems which the community wants to address without making it more complex:
OUTCOME: A follow on SWORD project has been funded by JISC (UK) along with a number of complementary (but separate) projects including DepositMO (http://blogs.ecs.soton.ac.uk/depositmo) and SONEX (http://sonexworkgroup.blogspot.com/).
Personally i'm involved in DepositMO which intends to use SWORD (+CRUD) at it's core and extend this even further (outside of SWORD) to be fully interactive with the users. More can be found on the levels of conformance via the DepositMO blog (http://blogs.ecs.soton.ac.uk/depositmo).
Package guidelines are to be set out by the new SWORD project along with tight definitions on what URIs mean and what it means to CRUD those URIs.
Being written in to both projects I hope to bring not only technical knowledge to the table but also real world usages.
There was also a call to look into technologies like OAuth and it's usages in SWORD, however this was a minor part of what became a major conversation at the second meeting.
Inverse Sword
This conversation started on workflows and a discussion on the opportunities for common workflows and their impact. The problem is that workflows tend to be very specific and quiet heavy weight in their approach to a problem, often constrained by the domain. This is the advantage of SWORD, it doesn't specify one, just a technique for transferring stuff. So what about reverse SWORD where you request a URI and a packaging format you want.
This basically then re-inforced the conversation on what it meant to have SWORD endpoints supporting full CRUD using content negotiation to agree on packaging formats. Clearly something to take forward... as it was!
Storage for Digital Repositories
Question was (not from me): What is their beyond the Akubra (now DuraCloud) and my two projects (one of which has been finished)?
It is clear that there are now a whole range of storage options and technologies with infinite numbers of APIs, luckily many of the cloud providers use the S3 API (which is good!). So what rules languages are there for expressing where things should be stored?
I briefly explained the EPrints implementation (labelled as mine but it isn't, it's EPrints property) which uses lightweight plug-ins to communicate with each service. These plug-ins implement 4 API calls (Store, Retrieve, Delete and one other necessary I won't bother explaining here). There is then an XML/XSLT based policy file which dictates which plug-ins are used to store what. Each file is then stored and metadata adjusted to state where it is stored in case policy changes. Upon a policy change, the files can be re-arranged to their correct locations again. This can also handle changes in storage architecture and whole services being off-lined. Advantage with this approach, which the community likes, is that you can use any number of storage solutions simultaneously and store as many copies of files on different ones as you like. For more see http://eprints.ecs.soton.ac.uk/17084/.
The actions from this were that others were going to look at this implementation to see if this rule based language could apply on other repository platforms. Further it would be nice to have some good reference architectures available from vendors.
Services and Configuration Languages (was Common Platforms/Tools on the day)
This was an interesting conversation which started around the idea of being able to re-use technologies by re-using/calling code libraries directly. The problem is here (as I see it) the number of coding environments and versions of these environments available.
The solution is REST (not SOAP) APIs on the web and abstraction APIs in the code (e.g. SOLR) which enable you to call functions from (say) the command line, without having to understand the code.
David Flanders perhaps summed it up best, there are levels of interaction, some easier than others:
There is no real answer to this question other than try and keep it simple... stupid.
Author IDs (URIs)
Yes it's our favourite topic raising its ugly head again!
It is clear that there are many efforts in this area, none of which have fully succeeded yet. There is still much interest in this area however and it is clear that we should be prepared to handle multiple IDs for a single author and be able to align them (if allowed) at a later stage.
Currently the project to watch is ORCID which is a continuation of a previous project by Thompson (which failed commercially in this project).
The consensus was however that we are not wrong to mint URIs for our authors in our repositories.
Conclusions
Identification/Authorisation is a problem, can technologies like OAuth not only help with authorisation but also with identification? This could be a very interesting area.
SWORD being taken forward is a very positive outcome of the first SITS meeting.
Simple services with simple APIs are so much more effective than "project centric" solutions and bloatware.
Simple services are usable by lots of people!
Anyway enough of that, what is actually happening?
Well I am very proud to be at the forefront of an international effort to hold a series of scholarly technology meetings focussed on solving institutional problems. These meetings, known as the Scholarly Information Technical Summit (SITS) meetings, are being held in alongside many international conferences over the next 2 years and are being backed by all the major international funding bodies. See http://bit.ly/Scholarly_Infrastructure_Technical_Summit for more info.
There have now been 2 meetings, although SITS only came about because the first one was so successful. Each meeting conforms to the Open Agenda (see wikipedia) principal and is chaired likewise. This leads to the agenda being very pertinent to the people in the room and often creates conversation critical to the forward momentum of some of the technologies discussed. In the next few paragraphs I'm going to try and summerise the hot topics from the first meeting:
SWORD - Put stuff in a repository
SWORD has undoubtedly been a huge success, it's simple and well supported by many publishers and publishing software (including most notably the Microsoft office suite via the author add-in tool http://research.microsoft.com/authoring). There are however some problems which the community wants to address without making it more complex:
- Packaging Formats - What exactly do you submit in your SWORD bundle, how should it be formed. There was no clear consensus other than we feel endpoints should try to support a multitude of formats depending on their users.
- Endpoints are hard to find, for both users and the software, this could do with being addressed either via negotiation or meta tags of some sort.
- URIs in the returned package are not well specified to say what they mean or what they should mean.
- Not a complete CRUD model
- No levels of compliance any more
- SWORD uses basic auth (too basic?)
OUTCOME: A follow on SWORD project has been funded by JISC (UK) along with a number of complementary (but separate) projects including DepositMO (http://blogs.ecs.soton.ac.uk/depositmo) and SONEX (http://sonexworkgroup.blogspot.com/).
Personally i'm involved in DepositMO which intends to use SWORD (+CRUD) at it's core and extend this even further (outside of SWORD) to be fully interactive with the users. More can be found on the levels of conformance via the DepositMO blog (http://blogs.ecs.soton.ac.uk/depositmo).
Package guidelines are to be set out by the new SWORD project along with tight definitions on what URIs mean and what it means to CRUD those URIs.
Being written in to both projects I hope to bring not only technical knowledge to the table but also real world usages.
There was also a call to look into technologies like OAuth and it's usages in SWORD, however this was a minor part of what became a major conversation at the second meeting.
Inverse Sword
This conversation started on workflows and a discussion on the opportunities for common workflows and their impact. The problem is that workflows tend to be very specific and quiet heavy weight in their approach to a problem, often constrained by the domain. This is the advantage of SWORD, it doesn't specify one, just a technique for transferring stuff. So what about reverse SWORD where you request a URI and a packaging format you want.
This basically then re-inforced the conversation on what it meant to have SWORD endpoints supporting full CRUD using content negotiation to agree on packaging formats. Clearly something to take forward... as it was!
Storage for Digital Repositories
Question was (not from me): What is their beyond the Akubra (now DuraCloud) and my two projects (one of which has been finished)?
It is clear that there are now a whole range of storage options and technologies with infinite numbers of APIs, luckily many of the cloud providers use the S3 API (which is good!). So what rules languages are there for expressing where things should be stored?
I briefly explained the EPrints implementation (labelled as mine but it isn't, it's EPrints property) which uses lightweight plug-ins to communicate with each service. These plug-ins implement 4 API calls (Store, Retrieve, Delete and one other necessary I won't bother explaining here). There is then an XML/XSLT based policy file which dictates which plug-ins are used to store what. Each file is then stored and metadata adjusted to state where it is stored in case policy changes. Upon a policy change, the files can be re-arranged to their correct locations again. This can also handle changes in storage architecture and whole services being off-lined. Advantage with this approach, which the community likes, is that you can use any number of storage solutions simultaneously and store as many copies of files on different ones as you like. For more see http://eprints.ecs.soton.ac.uk/17084/.
The actions from this were that others were going to look at this implementation to see if this rule based language could apply on other repository platforms. Further it would be nice to have some good reference architectures available from vendors.
Services and Configuration Languages (was Common Platforms/Tools on the day)
This was an interesting conversation which started around the idea of being able to re-use technologies by re-using/calling code libraries directly. The problem is here (as I see it) the number of coding environments and versions of these environments available.
The solution is REST (not SOAP) APIs on the web and abstraction APIs in the code (e.g. SOLR) which enable you to call functions from (say) the command line, without having to understand the code.
David Flanders perhaps summed it up best, there are levels of interaction, some easier than others:
- Core System (hard)
- Exposing structured data
- End user interfaces (including APIs)
There is no real answer to this question other than try and keep it simple... stupid.
Author IDs (URIs)
Yes it's our favourite topic raising its ugly head again!
It is clear that there are many efforts in this area, none of which have fully succeeded yet. There is still much interest in this area however and it is clear that we should be prepared to handle multiple IDs for a single author and be able to align them (if allowed) at a later stage.
Currently the project to watch is ORCID which is a continuation of a previous project by Thompson (which failed commercially in this project).
The consensus was however that we are not wrong to mint URIs for our authors in our repositories.
Conclusions
Identification/Authorisation is a problem, can technologies like OAuth not only help with authorisation but also with identification? This could be a very interesting area.
SWORD being taken forward is a very positive outcome of the first SITS meeting.
Simple services with simple APIs are so much more effective than "project centric" solutions and bloatware.
Simple services are usable by lots of people!
Monday, 5 October 2009
#ipres09: Are You Ready? Assessing Whether Organizations are Prepared for Digital Preservation.
Planets (mainly Tesella) did an online survey targetd at libraries adn archives in europe and then opened it up to 2000 others. Got 206 responses (70% european), 1/3 libraries, 1/4 archvies. Roles - 15% DP, 16% preservation in general, 22% in curation, 16% in IT, rest other.
Being a DP survey the majority said they were aware of DP issues. 1/6 haven't really thought that much about it and 1/2 don't have a DP policy. Interesting stat, if you have a DP policy in place your are 3 times more likely to have a budget in place. However the majority of those budgets are project budgets and not institutional budgets to actively perform DP in the organisation.
Also in the survey people are having to preserve all sorts of resources including databases.
Do people have control over the formats they can accept. National archives - Yes, National Archives (No). Others are pretty balanced between yes and no.
Amount of content: most say that they have less than 100Tb of content to preserve but people see it expanding to much larger quantities [when it won't be any more scary to preserve than it is now, we'll soon have 1Tb memory keys - heard it here first].
85% of people are trying to solve the problem and are looking to use plug-and-play components as people don't want to replace their current systems.
Most important thing in preservation is being able to maintain authenticity [possible european stance]. Emulation is less important. In the middle of the importance graph was the the importance of using metadata standards, but no one can decide what standards to use.
[Good survey, probably worth looking up and answering it yourself, especially if you are involved in a DP project! How do the aims of your project align with the view of the wider community?]
Being a DP survey the majority said they were aware of DP issues. 1/6 haven't really thought that much about it and 1/2 don't have a DP policy. Interesting stat, if you have a DP policy in place your are 3 times more likely to have a budget in place. However the majority of those budgets are project budgets and not institutional budgets to actively perform DP in the organisation.
Also in the survey people are having to preserve all sorts of resources including databases.
Do people have control over the formats they can accept. National archives - Yes, National Archives (No). Others are pretty balanced between yes and no.
Amount of content: most say that they have less than 100Tb of content to preserve but people see it expanding to much larger quantities [when it won't be any more scary to preserve than it is now, we'll soon have 1Tb memory keys - heard it here first].
85% of people are trying to solve the problem and are looking to use plug-and-play components as people don't want to replace their current systems.
Most important thing in preservation is being able to maintain authenticity [possible european stance]. Emulation is less important. In the middle of the importance graph was the the importance of using metadata standards, but no one can decide what standards to use.
[Good survey, probably worth looking up and answering it yourself, especially if you are involved in a DP project! How do the aims of your project align with the view of the wider community?]
#iPres09: An Emergent Micro-Services Approach to Digital Curation Infrastructure.
More stuff, smaller budget, more technology [good start]. We can gain though with digital tech through redundancy, meaning (through context), utility (through service) and value through use [totally agree with the last point esp. wrt OA publishing].
Projects imperatives: Do more with less, focus on the content, not the systems in which the content is managed [agree, focus on the data, not the software which people care less about]. This leads to the goal of the presentation which is micro-services, small services which provide maximum gain when they can be looped together, replaced, re-written etc.
There are 12 currently in 4 layers - Storage, Charecterisation/Inventory, Service (index, search, migration), Value (Usage) - [Good slide this] "LOCKSS, Lots of description keeps stuff meaningful, Lots of services keeps stuff useful, Lots of uses keeps stuff valuable".
Now describing some of the technologies which are being used. Simple restful api's to storage [however he hasn't mentioned the high level software they are using, based upon the first slide in the presentation said to use proven technology]. Emphasizing that storage has moved on (as it always will) and you can gain a lot of benefit though specialist products.
Storage is the main phase they have done and they have a few other mirco-services done. [Like the aim of it all however there are already lots of tools to provide each it is just the keeping them apart which is hard]
Projects imperatives: Do more with less, focus on the content, not the systems in which the content is managed [agree, focus on the data, not the software which people care less about]. This leads to the goal of the presentation which is micro-services, small services which provide maximum gain when they can be looped together, replaced, re-written etc.
There are 12 currently in 4 layers - Storage, Charecterisation/Inventory, Service (index, search, migration), Value (Usage) - [Good slide this] "LOCKSS, Lots of description keeps stuff meaningful, Lots of services keeps stuff useful, Lots of uses keeps stuff valuable".
Now describing some of the technologies which are being used. Simple restful api's to storage [however he hasn't mentioned the high level software they are using, based upon the first slide in the presentation said to use proven technology]. Emphasizing that storage has moved on (as it always will) and you can gain a lot of benefit though specialist products.
Storage is the main phase they have done and they have a few other mirco-services done. [Like the aim of it all however there are already lots of tools to provide each it is just the keeping them apart which is hard]
#iPres09: e-Infrastructure and digital preservation: challenges and outlook
e-infrastructure: Starts by defining infrastructure (see wikipedia) and e-infrastructure specific to a collection of European digital repositories. So basically we are looking at opportunities to build and supply services which are applicable to these repositories.
Background: EU is supplying lots of support for this and in germany they are researching national approaches, identifying activities and assign tasks to "expert" institutions. By introducing the current fields of project he is outlining that there is still a significant mismatch between the scale of the problem and the amount of effort being expended. From this he outlines that there is a significant lack of common approaches to solving problems. [I don't think this will ever go away, unless there is a mandate, and even then not everyone will want to sign up].
[Lots of argument] Funding is focused on many individual projects and thus doubles up the the argument that there are no commons. This led leads to a slide about interoperability and standards and the lack of them. [Which again, i don't think will ever go away and I think that we should be appreciative that people tend to pick XML to encode their data in, this makes it interoperable right].
[This is a start of project presentation, I don't seem to see that much output. They have some simple models as diagrams, again though at this stage it is hard to see how they are not just another project which will come up with (another) set of standards which no one will then want to adopt.]
Giving a set of examples now where they are going to re-use and extend existing software/projects. The goals are good, in terms of concrete steps for global infrastructure for registries, data formats, software deposits and risk management. [Just not sure how achievable all this is based upon the fact it has been the aim of many projects already]
Background: EU is supplying lots of support for this and in germany they are researching national approaches, identifying activities and assign tasks to "expert" institutions. By introducing the current fields of project he is outlining that there is still a significant mismatch between the scale of the problem and the amount of effort being expended. From this he outlines that there is a significant lack of common approaches to solving problems. [I don't think this will ever go away, unless there is a mandate, and even then not everyone will want to sign up].
[Lots of argument] Funding is focused on many individual projects and thus doubles up the the argument that there are no commons. This led leads to a slide about interoperability and standards and the lack of them. [Which again, i don't think will ever go away and I think that we should be appreciative that people tend to pick XML to encode their data in, this makes it interoperable right].
[This is a start of project presentation, I don't seem to see that much output. They have some simple models as diagrams, again though at this stage it is hard to see how they are not just another project which will come up with (another) set of standards which no one will then want to adopt.]
Giving a set of examples now where they are going to re-use and extend existing software/projects. The goals are good, in terms of concrete steps for global infrastructure for registries, data formats, software deposits and risk management. [Just not sure how achievable all this is based upon the fact it has been the aim of many projects already]
Tuesday, 8 September 2009
Thoughts on digitization, data deluge and linking
It's been a while since I've put a post up and this is probably due to being busy and also trying to tidy up a lot of stuff before starting on new projects.
In this post then: Digitisation
I never really gathered how big the area of digitisation is and how many non repository people are actively involved in digitisation. There are a great many projects >50 who are digitising resources and these include national libraries. Items being digitised include everything from postcards and newspapers to full books and old journals.
So what's the problem here ... simple ... how many people are digitising the same things? Yes I know that there is so much out there that this is unlikely to be the case however it brings me nicely to the problem of information overload. There is already more valuable information on the internet than we can possibly handle effectively, so how do you ensure that any resources you digitize for open access usage on the web can be found and used?
I don't normally say this but perhaps we should look at physical libraries for the answer. Libraries are a very good central point where you can find publications related to all subject areas, and if your local library does not have a copy then it will try and find a copy somewhere else.
How then does this map onto the web? Web sites become the library and links become the references to additional items or items this site does not contain, simple right? Unfortunately with 50+ projects I can count already, this leads to 50+ different web sites all with differing information presented in different ways. Due to the presentation of each web site being totally different this means that in fact they are not a library - that pride themselves on the standard way to organise resources -
thus web sites become books. Thus to find resources we have to rely on search engines and federation. Thus we are back to where we started and we have a problem with information overload.
Unfotunately I don't have an answer to this problem, however I do know that links hold the key to the solution. Each website at the moment is simply an island of infromation, what is desperately required is the authors and community to establish links to these resources. If digitisation houses are curating refereed resources then the simplist way to link to these would be to put information about them on wikipedia.
This would be my final point then, wikipedia is actually a good thing, simply because of the the community aspect. However it also provides many other huge benefits:
So if you are digitising books by an author, add this link to their wikipedia page. If you are digitising a collection of World War images, add links to some of these to wikipedia and flikr.
Establish links and help yourself to help everyone else.
In this post then: Digitisation
I never really gathered how big the area of digitisation is and how many non repository people are actively involved in digitisation. There are a great many projects >50 who are digitising resources and these include national libraries. Items being digitised include everything from postcards and newspapers to full books and old journals.
So what's the problem here ... simple ... how many people are digitising the same things? Yes I know that there is so much out there that this is unlikely to be the case however it brings me nicely to the problem of information overload. There is already more valuable information on the internet than we can possibly handle effectively, so how do you ensure that any resources you digitize for open access usage on the web can be found and used?
I don't normally say this but perhaps we should look at physical libraries for the answer. Libraries are a very good central point where you can find publications related to all subject areas, and if your local library does not have a copy then it will try and find a copy somewhere else.
How then does this map onto the web? Web sites become the library and links become the references to additional items or items this site does not contain, simple right? Unfortunately with 50+ projects I can count already, this leads to 50+ different web sites all with differing information presented in different ways. Due to the presentation of each web site being totally different this means that in fact they are not a library - that pride themselves on the standard way to organise resources -
thus web sites become books. Thus to find resources we have to rely on search engines and federation. Thus we are back to where we started and we have a problem with information overload.
Unfotunately I don't have an answer to this problem, however I do know that links hold the key to the solution. Each website at the moment is simply an island of infromation, what is desperately required is the authors and community to establish links to these resources. If digitisation houses are curating refereed resources then the simplist way to link to these would be to put information about them on wikipedia.
This would be my final point then, wikipedia is actually a good thing, simply because of the the community aspect. However it also provides many other huge benefits:
- External resources such as photoes have to have a licience
- In annotating a page/item you create links and establish facts which are available by semantic wikipedia (dbpedia)
- Wikipedia is an easy way to establish your presence on the link data web (linkeddata.org)
So if you are digitising books by an author, add this link to their wikipedia page. If you are digitising a collection of World War images, add links to some of these to wikipedia and flikr.
Establish links and help yourself to help everyone else.
Thursday, 7 May 2009
File Format Risk Analysis! How hard can it be?
The trouble with this is the sheer number of different formats out there (PDF,DOC,DOCX....) and the number of versions each of these has, each of which has it's own problems.
Enter the P2 data registry which now contains over 40,000 facts relating solely to format types. I have been working on this for a while now and a full list of services can be found on the registry homepage. Although the SPARQL Endpoint is a great place to start it doesn't help you much with risk analysis.
First thing is to determine which properties of a format or format super-type (PDF 1.6 has a super type PDF for example) are important when analyzing risk. Do you consider age, number of software tools, quality of documentation etc? And how do you add weightings to all of these properties as a factor of overall risk?
Using the registry as a starting point I have drawn up a profile involving these properties as well as many others which each return a high, medium or low risk level. These are then sum-mated and normalised to give a single risk score.
Enough of me talking: The live service (PDF 1.3) can be found here and here for PDF 1.4.
Wednesday, 8 April 2009
Less talk, less code, more data - The Preserv2 Data Registry
Yes, less talk more code (oxfordrepo.blogspot.com) is a good saying but i'm going to argue in this post that in fact we need more data! Having a ton of available services and a load of highly complex and well considered data models is all well and good but without data all of these services are useless; A repository is not a repository until it has something in it (Harnad).
If we look outside of the repository community for a minute we find the web community we are accumulating a whole ton of data, wikipedia being the main point of reference here. Yet in the repository community we are not harnessing this open linked data model to enhance our data.
I have been working in the area of digital preservation for a while now and the PRONOM file format registry (TNA UK) has been my friend for many years now and contains some valuable data. However I am concerned with the way I see it progressing. The main thing I use the PRONOM registry for is as a complement to DROID for file format information, and the data here is not even that complete. I am concerned however at the size of the new data model and the sheer effort which is going to be required to fill it with the data which it specifies.
Why not looked to the linked data web to see how to tie a series of smaller systems together to make a much more powerful and easier to maintain one!
This is where I have started with the preserv2 registry available at http://p2-registry.ecs.soton.ac.uk/.
The preserv2 registry is a semantic knowledge base (RDF triples based) with an SPARQL endpoint, RESTful services and a basic browser. Currently the data is focussed on file formats and is basically made up of the PRONOM database ported from a complex XML schema into simple RDF triples. On top of this i'm beginning to add data from dbpedia (wikipedia RDF'd) and making links between the PRONOM data and the dbpedia data!
Already this is helping is ascertain a greater knowledge base and the cost of gathering and compiling this data is very low. Other than that the registry took me less than a week to construct!
So "Go forth and make links" (Wendy Hall) is exactly what I'm now doing. With enough data you will be able to make complex OWL-S rules that can be used to deduce accurately facts such as formats which are at risk.
If we look outside of the repository community for a minute we find the web community we are accumulating a whole ton of data, wikipedia being the main point of reference here. Yet in the repository community we are not harnessing this open linked data model to enhance our data.
I have been working in the area of digital preservation for a while now and the PRONOM file format registry (TNA UK) has been my friend for many years now and contains some valuable data. However I am concerned with the way I see it progressing. The main thing I use the PRONOM registry for is as a complement to DROID for file format information, and the data here is not even that complete. I am concerned however at the size of the new data model and the sheer effort which is going to be required to fill it with the data which it specifies.
Why not looked to the linked data web to see how to tie a series of smaller systems together to make a much more powerful and easier to maintain one!
This is where I have started with the preserv2 registry available at http://p2-registry.ecs.soton.ac.uk/.
The preserv2 registry is a semantic knowledge base (RDF triples based) with an SPARQL endpoint, RESTful services and a basic browser. Currently the data is focussed on file formats and is basically made up of the PRONOM database ported from a complex XML schema into simple RDF triples. On top of this i'm beginning to add data from dbpedia (wikipedia RDF'd) and making links between the PRONOM data and the dbpedia data!
Already this is helping is ascertain a greater knowledge base and the cost of gathering and compiling this data is very low. Other than that the registry took me less than a week to construct!
So "Go forth and make links" (Wendy Hall) is exactly what I'm now doing. With enough data you will be able to make complex OWL-S rules that can be used to deduce accurately facts such as formats which are at risk.
Wednesday, 21 January 2009
EPrints 3.2 - Amazon S3/Cloudfront Plug-in
A quick post to say that we have just successfully tested an EPrints 3.2 (svn) install with the new Storage Controller plugged into Amazon S3!
This has quiet a lot of implications for both EPrints and other projects wanting to provide external services which operate on objects in a repository. We hope to bring people more news on this at the upcoming Open Repositories 2009 conference in Atlanta.
For more information on this all check out storage section on the Preserv2 website @ www.preserv.org.uk.
This has quiet a lot of implications for both EPrints and other projects wanting to provide external services which operate on objects in a repository. We hope to bring people more news on this at the upcoming Open Repositories 2009 conference in Atlanta.
For more information on this all check out storage section on the Preserv2 website @ www.preserv.org.uk.
Thursday, 18 September 2008
Institutions hate repositories... one simple reason.
Open access is not enough!
People want to give Open Access to some of their materials at their institution however the IR software is seen as a means to manage all Institutional content and not just that which is Open Access and part of the external image of the Institution.
The problem exists in the other direction as well where repository software is trying to solve these problems, thus people are not likely to use this software until it is included.
So what do we end up with...
Lots of Repository Islands which aren't interoperable with each other!
So if we solve the access and copyright issue will people use the software? errrr No. At this point the software is an all in solution and not a service which can be utilised by current institutional practise ... Give up...?
No!
Focus on providing a service, e.g. something which can manage your Digital Resources and enable this to plug to existing institutional services. Some softwares would argue they support this already. OK good, so don't try and solve the problem if it is just an integration issue.
To the repositories: Decouple! Build a set of services, build ways of plugging services together and allow the community to pic 'n' mix.
To the institution: You already have access control systems ask your Information/Computer Systems department. You probably already have a Content Management System for educational resources for students (Blackboard? - Integrates with an LDAP server), these use external services to manage access and authentication! Here's a few services for you... LDAP, Radius, Eduroam, Domain Controller.
Till next time!
People want to give Open Access to some of their materials at their institution however the IR software is seen as a means to manage all Institutional content and not just that which is Open Access and part of the external image of the Institution.
The problem exists in the other direction as well where repository software is trying to solve these problems, thus people are not likely to use this software until it is included.
So what do we end up with...
Lots of Repository Islands which aren't interoperable with each other!
So if we solve the access and copyright issue will people use the software? errrr No. At this point the software is an all in solution and not a service which can be utilised by current institutional practise ... Give up...?
No!
Focus on providing a service, e.g. something which can manage your Digital Resources and enable this to plug to existing institutional services. Some softwares would argue they support this already. OK good, so don't try and solve the problem if it is just an integration issue.
To the repositories: Decouple! Build a set of services, build ways of plugging services together and allow the community to pic 'n' mix.
To the institution: You already have access control systems ask your Information/Computer Systems department. You probably already have a Content Management System for educational resources for students (Blackboard? - Integrates with an LDAP server), these use external services to manage access and authentication! Here's a few services for you... LDAP, Radius, Eduroam, Domain Controller.
Till next time!
Saturday, 19 July 2008
Is winning in Casino's a Bad Thing
Totally off my usual topics but by playing short games in the Casinos in Vegas and quitting while ahead i'm up by just over $100. It's not a lot but considering i've only been playing 1c machines and $5 - $10 blackjack I think that's quiet cool! However since I haven't lost yet does that mean i'll now want to continue... could it get addictive. Considering i'm going to Atlantic City next week is this bad news!
As for the technical note, follow the rules of blackjack on wikipedia (http://en.wikipedia.org/wiki/Blackjack) and make sure you buy in enough for 10-12 hands at minimum bet, and never bet more unless wikipedia says you should! Also when you are up, 3-4 decks worth of rounds... get out!
This may not be the longest game in the world but you take the money off the Casino!
Thank-you Venitian/Palazzo Las Vegas!
As for the technical note, follow the rules of blackjack on wikipedia (http://en.wikipedia.org/wiki/Blackjack) and make sure you buy in enough for 10-12 hands at minimum bet, and never bet more unless wikipedia says you should! Also when you are up, 3-4 decks worth of rounds... get out!
This may not be the longest game in the world but you take the money off the Casino!
Thank-you Venitian/Palazzo Las Vegas!
Wednesday, 16 July 2008
#crigshow - Conference 2 - Worldcomp
Agents and Web Services... Why no collaboration?
Out of all the presentations at worldcomp this one struck me as one of the most obvious but not covered areas for research in computer science. Probably the most well known agent system is that used by the travel industry where they have standard ways of interfacing with each other to find details of travel and hotels available on a global scale. This is no mean feat with the number of companies there are hooking into this network.
So why doesn't the same exist for web services or if there is such a system why isn't everyone in the open community using it?
Surely the point of web services is for people to discover and use them in their own scenarios just like the agents in the travel industry do. OK so maybe the problem lies in the fact that there are so many communities that there will never be a specific use case or framework and thus hosting a generic web service network becomes infinitely hard with the number of different APIs and Implementations.
OK so if you are going to use Agents in Web Services what issues do you need to consider? Also what do you gain through doing this?
One of the key ideas which came out of a talk at worldcomp is to use Agents to be the intelligent front to a web service. This enables an agent to track of a set of web services including information about a specific web service such as availability, versions, changing cost and and offline copy if the service allows this. So the agent becomes a Rendezvous Point for a series of web services.
So why aren't we seeing more collaboration between the Agent community and the Web Services community?
Out of all the presentations at worldcomp this one struck me as one of the most obvious but not covered areas for research in computer science. Probably the most well known agent system is that used by the travel industry where they have standard ways of interfacing with each other to find details of travel and hotels available on a global scale. This is no mean feat with the number of companies there are hooking into this network.
So why doesn't the same exist for web services or if there is such a system why isn't everyone in the open community using it?
Surely the point of web services is for people to discover and use them in their own scenarios just like the agents in the travel industry do. OK so maybe the problem lies in the fact that there are so many communities that there will never be a specific use case or framework and thus hosting a generic web service network becomes infinitely hard with the number of different APIs and Implementations.
OK so if you are going to use Agents in Web Services what issues do you need to consider? Also what do you gain through doing this?
One of the key ideas which came out of a talk at worldcomp is to use Agents to be the intelligent front to a web service. This enables an agent to track of a set of web services including information about a specific web service such as availability, versions, changing cost and and offline copy if the service allows this. So the agent becomes a Rendezvous Point for a series of web services.
So why aren't we seeing more collaboration between the Agent community and the Web Services community?
Monday, 14 July 2008
#crigshow - Conference 1 - Oscelot
This open source day (#osdiii) hosted by Oscelot was an unconferene which soon became based heavily around the Blackboard platform. This was expected as the majority of people attending it were then going on to attend the BbWorld conference. With the title of the conference being Open Source and yet the main topic being that of a Closed Source product this gave an opening for the CRIG team to promote the wider Open Source community to those who are focused on Blackboard use cases.
The day was a success for the team as we promoted good practices in web development, standards, resource management and the fact that the people who manage an eLearning platform has a responsibility to the content they hold.
From our point of view, we discovered: If blackboard is the industry leader in learning management systems then the repository community is big problems when it comes to archiving these resources by the current methodologies each community practices.
More Collaboration and Awareness please!
The day was a success for the team as we promoted good practices in web development, standards, resource management and the fact that the people who manage an eLearning platform has a responsibility to the content they hold.
From our point of view, we discovered: If blackboard is the industry leader in learning management systems then the repository community is big problems when it comes to archiving these resources by the current methodologies each community practices.
More Collaboration and Awareness please!
Subscribe to:
Posts (Atom)