Hi there tout la gang,

We don't have much to say about research in practice at the Café right now

but we are talking policy and practice over here now: Literacy Enquirers.

Wednesday, March 11, 2009

it is all fun and games until...

The Canadian Council on Learning has posted their online literacy assessment tool.

The tool can be used to assess your IALSS level. IALSS is the International Adult Literacy and Skills Survey:

"The International Adult Literacy and Skills Survey was a seven-country initiative conducted in 2003. In every country nationally representative samples of adults aged 16-65 were interviewed and tested at home, using the same psychometric test to measure prose and document literacy as well as numeracy and problem-solving skills.
  • ...The main purpose of the survey was to find out how well adults used printed information to function in society.
  • Another aim was to collect data on the incidence and volume of participation in adult education and training, and to investigate the relationships between initial and adult education, on the one hand, and literacy, numeracy and problem-solving proficiency and wider economic and social outcomes, on the other.
  • In addition, a subsidiary goal was to provide information regarding change in the distribution of skills over the years since the previous survey (the 1994, International Adult Literacy Survey).
  • ...The data are used to inform policy decisions, help effectively allocate resources where needed and inform decisions on the composition and content of remedial skill development course and adult education."

Ever since the first IALS survey in 1994, there has been debate* about what exactly these data tell us, but, until now, I do not think that anyone has suggested that an IALSS-based test, designed to survey a population, ever be used to assess and assign levels to individuals. If they have, they should be thoroughly ashamed of themselves.

A couple of us tried the test. It took ages, the fonts were teeny weeny, the interface did not work in any browser, and it was incredibly dull. I think that the test could take quite a long time to complete. We did not want to spend too long on it so we tried not to get bogged down. We just clicked on our first good idea for the right answer and let it go.

We both scored Level 4 (Strong skills -- individuals at these levels can process information of a complex and demanding nature) in Document Literacy (The ability to find and use information in forms, charts, graphs and other tables).

My colleague scored Level 4 in Numeracy (The ability to use basic math skills in everyday life) and I scored Level 3 (Adequate to cope with the demands of everyday life and work in an advanced society -- roughly denotes the skill level required for successful high school completion and college entry). I must admit I did not feel like doing some of the calculations and guessed at some of the answers but Level 3 is what I would guess is correct for me in any case and, as it turns out, I am a pretty good guesser.

Now the bad news. We both scored Level 2 (A capacity to deal only with simple, clear material involving uncomplicated tasks -- people at this level may develop everyday coping skills, but their poor literacy skills make it hard to conquer challenges such as learning new job skills) in Prose Literacy (The knowledge and skills required to understand and appropriately use information from print materials). CCL recommends that all Canadians need to be at Level 3 if they are to achieve prosperity, health and happiness.

My colleague is very clever and has exemplary prose literacy skills. I am using this fact to cheer myself up a little. I am surprised to find that despite my rather cavalier attitude to this test, the result made me feel a little crappy. I usually feel fine about my ability to understand and appropriately use information from print materials. There are things that challenge me but I have confidence that if I persevere I will get it.

Our Level 2 scores prompted another colleague to wonder, "If the CCL online test is based on or reflects the IALSS test and if [these two] only scored 2 on the Prose test, perhaps the IALSS test is underestimating people’s literacy levels."

One of the questions that has been raised about IALS and IALSS is why the results show such high numbers of people at Level 1 and 2 when people self-assess at a much greater ability to understand and appropriately use information from print materials.

But the biggest question for me remains, “Why do I want to know what IALSS level I am at?” or “Why does anyone want to know what IALSS level any individual is at?”

What we want to know is how do we work with text now and how do we want or need to be able to work with text to do what we want or need to do. To do this we can use a method such as the Canadian Adult Reading Assessment that assesses reading skills and strategies in a manageable, non-invasive, non-judgmental way and demonstrates how learner-centred assessment leads to the development of a learner-centred curriculum that is relevant and effective.

C'mon CCL, stop now before somebody gets hurt.

*For some of that debate see Literacies Issue #2 and the Literacies IALS web forum.


Anonymous said...

HA! I'm stuck on the 1st question!

It asks "Which locations allow cats?" What it meant to ask was "Which of the following locations allow cats?" The difference being that their question asks for ALL the cat-safe locations. Since none of the answers they give to choose from list all the locations... the best we can do is sort out which combination of addresses includes only cat-safe homes.

Working with GED materials - and trying to create GED type materials - has taught me to parse questions very carefully. It's all too easy to make a question ambiguous or inexact.

I'll see if I can finish the test and let you know how I do.

Wendell Dryden said...

Hey! I scored 3, 4 and 4. Clearly this is a very perceptive and accurate test!

So... um... What's the 'literacy' dimension to the question "Which snack do you think is chicken?"

Other complaints: One question repeated the same choice twice (first and fourth answer options). I disagreed with their interpretation of one bit of Carroll (but knew enough to find the answer they likely intended). I wish I could see which ones they marked incorrect.

I'm guessing I found this quiz fairly easy exactly because of it's familiar, schoolish nature. I spend forever helping people develop strategies for test taking and multiple-choice guessing. I fill out lots of forms. And I recognize those hackeyed math word problems about unit price or grams of salt or the cheapest bed - not from my grown-up life as a shopper, but from my day job teaching math.

literacies publisher said...

I always knew it! You are the smartest literacy worker EVER!

Well then according to this handy dandy little chart -- http://www.ccl-cca.ca/LiteracyAssessment2009/LiteracyCanada.html#income -- you should be making a cool $60,782.03 compared to my paltry Level 2 salary of $36,095.91.

Good point about the cats. And yes the 'Which is chicken?' question had us stumped. As both respondents and assessors.

And I though the whole point of Alice was not that she was dreaming, but that Carroll was stoned. In a good way.

Don't bogart that joint my friend, I have to find out my IALSS level again...

Anonymous said...

Oh, I'm glad I'm not the only one who was surprised by some of my results! I scored 3 on both Document and Prose Literacy. If literacy workers are averaging 2s and 3s in these sections, I'm thinking something's not quite right here.

I also noticed several problems with the wording of questions, as well as answer choices being repeated (I have contacted CCL about the latter).

Also, I'd just like to vent about this one: "Which snack has the least calories per gram?" ...It should be FEWEST, not LEAST. Eesh!

Anonymous said...
This comment has been removed by a blog administrator.
Judi said...

Well, I finally did the test. I got 4 prose, 3 document and 5 numeracy (hope everyone is impressed).

I feel like a lot of the questions were actually measuring my level of cynicism, which is perhaps and essential skill for literacy workers.

How do I know why the kid kept the change? How do I know whether the cafe was really trying to get customer's contact information?

The other thing that's a bit disconcerting is the "margin of error." Apparently, my 75% in Document use is really somewhere between 53 - 90%? I think that's what is says, but I may be wrong (I'll defer to those of you who got 4 in document use).

In the meantime, I'll just keep telling myself that my 3 is really a 5.


Post a Comment