Adventures & experiences in contemporary technology
In the early years of my SAS programming career, there seemed to be just the one procedure to use if you wanted to create a summary table. That was TABULATE and it was great for generating columns according to the value of a variable such as the treatment received by a subject in a clinical study. To a point, it could generate statistics for you too and I often used it to sum frequency and percentage variables. Since then, it seems to have been enhanced a little and it surprised me with the statistics it could produce when I had a recent play. Here’s the code:
proc tabulate data=sashelp.class;
class sex;
var age;
table age*(n median*f=8. mean*f=8.1 std*f=8.1 min*f=8. max*f=8. lclm*f=8.1 uclm*f=8.1),sex / misstext="0";
run;
When you compare that with the idea of creating one variable per column and then defining them in PROC REPORT as many do, it has to look more elegant and the results aren’t bad either though they can be tweaked further from the quick example that I generated. That last comment brings me to the point that PROC REPORT seems to have taken over from TABULATE wherever I care to look these days and I do ask myself if it is the right tool for that for which it is being used or if it is being used in the best way.
Using Data Step to create one variable per column in a PROC REPORT output doesn’t strike me as the best way to write reusable code but there are ways to make REPORT do more for you. For example, by defining GROUP, ACROSS and ANALYSIS columns in an output, you can persuade the procedure to do the summarising for you and there’s some example code below with the comma nesting height under sex in the resulting table. Sums are created by default if you do this and forgoing an analysis column definition means that you get a frequency table, not at all a useless thing in many cases.
proc report data=sashelp.class nowd missing;
columns age sex,height;
define age / group "Age";
define sex / across "Sex";
define height / analysis mean f=missing. "Mean Height";
run;
For those times when you need to create more heavily formatted statistics (summarising range as min-max rather showing min and max separately, for example), you might feel that the GROUP/ACROSS set-up’s non-display of character values puts a stop to using that approach. However, I found that making every value combination unique and attaching a cell ID helps to work around the problem. Then, you can create a format control data set from the data like in the code below and create a format from that which you can apply to the cell ID’s to display things as you need them. This method does make things more portable from situation to situation than adding or removing columns depending on the values of a classification variable.
proc sql noprint;
create table cntlin as
select distinct "fmtname" as fmtname, cellid as start, cellid as end, decode as label
from report;
quit;
proc format lib=work cntlin=cnlin;
run;
This is one of the classic SAS errors that you can get from PROC SQL and it can be thrown by a number of things. Missing out a comma in a list of variables on a SELECT statement is one situation that will do it, as will having an extraneous one. As I discovered recently, an ill-defined SAS function nesting like LEFT(TRIM(PERIOD,BEST.)) will have the same effect; notice the missing PUT function in the example. The latter surprised me because I might have expected something more descriptive for this as would be the case in data step code. In the event, it took some looking before the problem hit me because it’s amazing how blind you can become to things that are staring you in the face. Familiarity really can make you pay less attention.
This could be the beginning of a series on error messages from PROC SQL that may appear unclear to a programmer more familiar with Data Step. The cause of my getting the message that heads this posting is that there was a numeric variable with a length less that the default of 8, not the best of situations. Sadly, the message doesn’t pin point the affected variable so it took some commenting out of pieces of code before I found the cause of the problem. That’s never to say that PROC SQL does not have debugging functionality in the form of FEEDBACK, NOEXEC, _METHOD and _TREE options on the PROC SQL line itself or the validation statement but neither of these seemed to help in this instance. Still, they’re worth keeping in mind for the future as is SAS Institute’s own page on SQL query debugging. Of course, now that I know what might be the cause, a simple PROC SQL report using the dictionary tables should help. The following code should do the needful:
proc sql;
select memname, name, type, length
from dictionary.columns
where libname="DATA" and type="num" and length ne 8;
quit;
Recently, I had the opportunity to have a look at the Excel library engine again because I need to read Excel data into SAS. You need SAS Access for PC Files licensed for it to but it does simplify the process of getting data from spreadsheets into SAS. It all revolves around setting up a library pointing at the Excel file using the Excel engine. The result is that every worksheet in the file is treated like a SAS dataset even if there names contain characters that SAS considers invalid for dataset names. The way around that is to enclose the worksheet name in single quotes with the letter n straight after the closing quote, much in the same way as you’d read in text strings as SAS date values (’04MAR2010’d, for example). In order to make all of this more, I have added some example code below.
libname testxl excel 'c:\test.xls';
data test;
set testxl.'sheet1$'n;
run;
All of the above does apply to SAS on Windows (I have used it successfully in 9.1.3 and 9.2) but there seems to be a way of using the same type of thing on UNIX too. Again, SAS Access for PC Files is needed as well as a SAS PC Files server on an available Windows machine and it is the PCFILES engine that is specified. While I cannot say that I have had the chance to see it working in practice but seeing it described in SAS Online Documentation corrected my previous misimpressions about the UNIX variant of SAS and its ability to read in Excel or Access data. Well, you learn something new every day.
There are whole pile of SAS functions for testing text strings that hadn’t come to my attention until this week. Until then, I’d have gone about using functions like INDEX and PRXMATCH functions for the same sort of ends but it’s never any load to have a few different ways of doing things and to use the right one for the job. Here’s a quick list of my recent discoveries:
ANYALNUM: First position of any alphanumeric character, returns 0 if absent
ANYALPHA: First position of any alphabetic character (letter of the alphabet), returns 0 if absent
ANYCNTRL: First position of any control character, returns 0 if absent
ANYDIGIT: First position of any numeric character, returns 0 if absent
ANYFIRST: First position of any character that can be used as the start of a SAS variable name when VALIDVARNAME is set to V7, returns 0 if absent
ANYGRAPH: First position of any printable character that isn’t white space, returns 0 if absent
ANYLOWER: First position of any lowercase letter, returns 0 if absent
ANYNAME: First position of any character that can be used in a SAS variable name when VALIDVARNAME is set to V7, returns 0 if absent
ANYPRINT: First position of any printable character, returns 0 if absent
ANYPUNCT: First position of any punctuation character, returns 0 if absent
ANYSPACE: First position of any whitespace character (tabs, carriage returns and the like), returns 0 if absent
ANYUPPER: First position of any uppercase letter, returns 0 if absent
ANYXDIGIT: First position of any hexadecimal character, returns 0 if absent
NOTALNUM: First position of any non-alphanumeric character, returns 0 if absent
NOTALPHA: First position of any non-alphabetic character, returns 0 if absent
NOTCNTRL: First position of anything that isn’t a control character, returns 0 if absent
NOTDIGIT: First position of any non-numeric character, returns 0 if absent
NOTFIRST: First position of any character that cannot be used as the start of a SAS variable name when VALIDVARNAME is set to V7, returns 0 if absent
NOTGRAPH: First position of anything that isn’t a printable character that isn’t white space, returns 0 if absent
NOTLOWER: First position of anything that isn’t a lowercase letter, returns 0 if absent
NOTNAME: First position of any character that cannot be used in a SAS variable name when VALIDVARNAME is set to V7, returns 0 if absent
NOTPRINT: First position of any non-printable character, returns 0 if absent
NOTPUNCT: First position of anything that isn’t a punctuation character, returns 0 if absent
NOTSPACE: First position of anything that isn’t a whitespace character, returns 0 if absent
NOTUPPER: First position of anything that isn’t an uppercase letter, returns 0 if absent
NOTXDIGIT: First position of anything that isn’t a hexadecimal character, returns 0 if absent
Apart from simpler cases where other techniques would work well with the a similar amount of effort, there are others that would need some investigation if you were program them without using one of the above functions. For that reason, I’ll be keeping them in mind for when I might meet one of those more complex scenarios.
Recently, I was trying something like this and noted some odd behaviour:
data _null_;
file fileref;
put "text %sysfunc(pathname(work)) more text";
run;
This is the kind of thing that I was getting:
text c:\sasworkmore text
In other words, the space after %sysfunc was being ignored and, since I was creating and executing a Windows batch file using SAS 8.2, the command line action wasn’t doing what was expected. The fix was simple but I reckoned that I’d share what I saw anyway, in case it helped anyone else:
data _null_;
file fileref;
x="text %sysfunc(pathname(work))"||" more text";
put x;
run;
The inspiration for this post is a situation where having the string “OR” or “AND” as an input to a piece of SAS Macro code breaking a program that I had written. Here is a simplified example of what I was doing:
%macro test;
%let doms=GE GT NE LT LE AND OR;
%let lv_count=1;
%do %while (%scan(&doms,&lv_count,' ') ne );
%put &lv_count;
%let lv_count=%eval(&lv_count+1);
%end
%mend test;
%test;
The loop proceeds well until the string “AND” is met and “OR” has the same effect. The result is the following message appears in the log:
ERROR: A character operand was found in the %EVAL function or %IF condition where a numeric operand is required. The condition was: %scan(&doms,&lv_count,' ') ne
ERROR: The condition in the %DO %WHILE loop, , yielded an invalid or missing value, . The macro will stop executing.
ERROR: The macro TEST will stop executing.
Both AND & OR (case doesn’t matter but I am sticking with upper case for sake of clarity) seem to be reserved words in a macro DO WHILE loop while equality mnemonics like GE cause no problem. Perhaps, the fact that and equality operator is already in the expression helps. Regardless, the fix is a simple one:
%macro test;
%let doms=GE GT NE LT LE AND OR;
%let lv_count=1;
%do %while ("%scan(&doms,&lv_count,' ')" ne "");
%put &lv_count;
%let lv_count=%eval(&lv_count+1);
%end
%mend test;
%test;
Now none of the strings extracted from the macro variable &DOMS will appear as bare words and confuse the SAS Macro processor but you do have to make sure that you are testing for the null string (“” or ”) or you’ll send your program into an infinite loop, always a potential problem with DO WHILE loops so they need to be used with care. All in all, an odd looking message gets an easy solution without recourse to macro quoting functions like %NRSTR or %SUPERQ.
I ran into some trouble with creating new templates and styles while running a SAS job in batch mode. The cause was the user of the RSASUSER switch on the SAS command. This sets the SASUSER library to be read-only and that is what is used to store new templates and styles by default. The fix is to switch these to another library to which there is write access, WORK, for example. Here’s the line of code that achieves the manoeuvre:
ODS PATH work.templat(update) sasuser.templat(read) sashelp.tmplmst(read);
Apparently, the same change might be needed for stored processes too so it’s one to keep in mind.
Recently, I tried code like this in a SAS macro:
data sections;
infile datalines dlm=",";
input graph_table_number $15. text_line @1 @;
datalines;
"11.1 ,Section 11.1",
"11.2 ,Section 11.2",
"11.3 ,Section 11.3"
;
run;
While it works in its own right, including it as part of a macro yielded this type of result:
ERROR: The macro X generated CARDS (data lines) for the DATA step, which could cause incorrect results. The DATA step and the macro will stop executing.
A bit of googling landed me on SAS-L where I spotted a solution like this one that didn’t involving throwing everything out:
filename temp temp;
data _null_;
file temp;
put;
run;
data sections;
length graph_table_number $15 text_line $100;
infile temp dlm=",";
input @;
do _infile_=
"11.1 ,Section 11.1",
"11.2 ,Section 11.2",
"11.3 ,Section 11.3"
;
input graph_table_number $15. text_line @1 @;
output;
end;
run;
filename temp clear;
The filename statement and ensuing data step creates a dummy file in the SAS work area that gets cleared at the end of every session. That seems to fool the macro engine into thinking that input is from a file and not the CARDS/DATALINES method to which it takes grave exception. The trailing @’s hold an input record for the execution of the next INPUT statement within the same iteration of the DATA step so that the automatic variable _infile_ can be fed as part of the input process in a do block with the output statement ensure that new records from the input buffer reach the data set being created.
This method does work but I would like to know the underlying reason as to why SAS Macro won’t play well with included data entry using DATALINES or CARDS statements in a data step, particularly when it allows other methods that using either SQL insert statements or standard variable assignment in data step. I find it such a curious behaviour that I remain on the lookout for the explanation why it is like this.
In a world where write access to a folder or directory is controlled by permission settings at the operating system level, a ready answer for when you get the above message in your log when creating a SAS data set would be to check your access. However, if you are working in Windows and your access seems fine, then SAS’ generation of an access error message seems all the more perplexing.
Unlike the more black-and-white world of UNIX and Linux, Windows has other ways to change access that could throw things off from the straight and narrow. One of them, it would appear, is to right-click on the file listing pane in Windows Explorer and select “Customize this folder…” to change how it appears. The strange upshot of this is that a perpetual read-only flag is set for the folder in question and that flag triggers SAS authorisation errors. The behaviour is very strange and unexpected when you find it, and the quickest and easiest solution sounds drastic. This involves deleting the folder and creating a new one in its place, saving anything that you want to retain in another temporary location. An alternative approach uses the attrib command and is less invasive.
It begs the question as to why Microsoft is re-appropriating a flag used for access purposes to be used to determine whether the HTML components of a folder display have been changed or not. This is very strange stuff and does not look like good software design at all. With all the other problems Microsoft creates for itself, I am not holding my breath until it’s fixed either. There seem to be other things like this waiting to catch you out when using Windows SAS and a good place to start is with SAS’ own description of the problem that I have just shared.