Showing posts with label load. Show all posts
Showing posts with label load. Show all posts

Tuesday, March 27, 2012

Deadlocks

Our system is reasonably complex with a lot of non-trivial stored procedures. As the load on our DB increased we're now getting more and more deadlocks (10 per day or so from about a million stored proc executions).

We try to avoid transactions where we can, and we do attempt to optimse stored procs to steer clear of deadlock conditions, but with the sheer number of stored procedures we can't possibly avoid all deadlock conditions.

One solution I'm considering is to re-run stored procs that failed because of a deadlock. In the .net code we'll run the stored proc, check for a deadlock error and if one happened, wait 100ms and try again.

What do you guys think?SQL-Server-Performance.com is a site that I use often to help me with situations such as this.

They have an article calledTips for Reducing SQL Server Deadlocks which I highly recommend. Part of that article states "Most well-designed applications, after receiving a deadlock message, will resubmit the aborted transaction, which most likely can now run successfully.", which is exactly what you are proposing to do so it is sounding like a good idea.

These 3 suggestions have all but eliminated deadlocking for me:
-- Keep transactions as short as possible
-- Reduce lock time
-- Consider using the NOLOCK hint

Terri|||Thanks, that's what I wanted to see.

Our solution then:

- All stored procedures with no updates / deletes will use SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED

- I'll overload the SqlCommand.ExecuteNonQuery() method to re-run deadlocked transactions

- In transactions doing updates / deletes we'll use the NOLOCK hint where serializability is not required

in addition to our current practices:

- Only use transactions where serializability is required

- Keep the transaction as short as possible

- Optimise select statements in transactions with indexes on non-trivial tables

Cheers for that link Terri|||I would definitely use NOLOCKs on any queries that "dirty reads" are ok. Also, you can use WITH (ROLOCK) on update and delete statements where you are deleting one row. Such as deleting based on a Primary key.|||Do you know what type of Deadlocks you're are getting, lock promotion deadlocks or the more traditional resource contention type?|||Just regular resource contention ones.|||Pierre, Can you give an example of how you ended up re-running the transaction? I am having the same problems, I get about 5-10 per day also. Any help would be appriciated!
Thanks,
Jason|||

Try the code below it is what is recommended by my book SQL Server 2000 A beginner's guide by Dusan Petkovic. But this code is for SQL Server 2005 so test it. The code is from the link below. The key is to write a conditional statement that will return SQL Server @.@. ERROR 1205 which is Deadlock. Run a search for SET DEADLOCK_PRIORITY in the BOL (books online). Hope this helps.
CREATE PROCEDURE DeadLock_Test AS

SET NOCOUNT ON
SET XACT_ABORT ON
SET DEADLOCK_PRIORITY LOW

DECLARE @.Err INTEGER
DECLARE @.ErrMsg VARCHAR(200)

RETRY:
BEGIN TRY
BEGIN TRANSACTION
UPDATE tblContact SET LastName = 'SP_LastName_1' WHERE ContactID = 1
UPDATE tblContact SET LastName = 'SP_LastName_2' WHERE ContactID = 2
COMMIT TRANSACTION
END TRY
BEGIN CATCH
SET @.Err = @.@.ERROR
IF @.Err = 1205
ROLLBACK TRANSACTION
INSERT INTO ErrorLog (ErrID, ErrMsg) VALUES (@.Err, 'Deadlock recovery attempt.')
WAITFOR DELAY '00:00:10'
GOTO RETRY
IF @.Err = 2627
SET @.ErrMsg = 'PK Violation.'
IF @.ErrMsg IS NULL
SET @.ErrMsg = 'Other Error.'
INSERT INTO ErrorLog (ErrID, ErrMsg) VALUES (@.Err, @.ErrMsg)
END CATCH
http://www.campbellassociates.ca/blog/CategoryView.aspx?category=SQL%20Server

|||

Pierre,

I finally found my error now after a year, when they say use Query Analyizer They mean it. I had a stupid trigger that I had written way before I knew what I was doing (I still don't) but anyway the trigger was poorly written and actuall not needed. No More Deadlocks!!! Weheww!!! So I think the moral to the story is don't use Triggers Unless you absolutely have to.

sql

Deadlocking Limitations of SQL Server... tell me it isn't so.

Hi,
I have a client-server .NET system that uses an Enterprise Services
Serviced Component (COM+ component) for data access. Under high load,
I am getting deadlocking errors, they seem to be related to one table.
These situations are hard to debug, but I am guessing it is because
an update on a delete may be occurring on DIFFERENT ROWS in the same
table at the same time. This can't be right, can it?
I read something about problems when using indexes, but this table is
not indexed other than the primary key. The table definition is shown
below. Any suggestions would be appreciated.
Thanks!
*** TABLE DEFINITION (FIELDS HAVE BEEN RENAMED, BUT STRUCTURE IS
ACCURATE) ***
CREATE TABLE [Boo_Record_Foo] (
[Boo_Id] [int] NOT NULL ,
[Fooed_By_User_Id] [int] NULL ,
[Fooed_By_User_Name] [varchar] (30) COLLATE
SQL_Latin1_General_CP1_CI_AS NULL ,
CONSTRAINT [PK_Boo_Record_Foo] PRIMARY KEY CLUSTERED
(
[Boo_Id]
) WITH FILLFACTOR = 90 ON [PRIMARY]
) ON [PRIMARY]
GO
*** ERROR MESSAGE ***
Transaction (Process ID 53) was deadlocked on {lock} resources with
another process and has been chosen as the deadlock victim. Rerun the
transaction.COM+ tends to use the SERIALIZED isolation level which is never good for
multi-user apps. I would check to see what the isolation level is on all
the connections. You say your table has no index other than the PK
constraint. Is it ever accessed by anything other than the PK? Can you
show the 2 statements that are being used when it deadlocks?
--
Andrew J. Kelly
SQL Server MVP
"Don MacKenzie" <cd_mackenzie@.hotmail.com> wrote in message
news:2544f4a.0402131647.7bbd58cf@.posting.google.com...
> Hi,
> I have a client-server .NET system that uses an Enterprise Services
> Serviced Component (COM+ component) for data access. Under high load,
> I am getting deadlocking errors, they seem to be related to one table.
> These situations are hard to debug, but I am guessing it is because
> an update on a delete may be occurring on DIFFERENT ROWS in the same
> table at the same time. This can't be right, can it?
> I read something about problems when using indexes, but this table is
> not indexed other than the primary key. The table definition is shown
> below. Any suggestions would be appreciated.
> Thanks!
> *** TABLE DEFINITION (FIELDS HAVE BEEN RENAMED, BUT STRUCTURE IS
> ACCURATE) ***
> CREATE TABLE [Boo_Record_Foo] (
> [Boo_Id] [int] NOT NULL ,
> [Fooed_By_User_Id] [int] NULL ,
> [Fooed_By_User_Name] [varchar] (30) COLLATE
> SQL_Latin1_General_CP1_CI_AS NULL ,
> CONSTRAINT [PK_Boo_Record_Foo] PRIMARY KEY CLUSTERED
> (
> [Boo_Id]
> ) WITH FILLFACTOR = 90 ON [PRIMARY]
> ) ON [PRIMARY]
> GO
>
> *** ERROR MESSAGE ***
> Transaction (Process ID 53) was deadlocked on {lock} resources with
> another process and has been chosen as the deadlock victim. Rerun the
> transaction.|||Hi Don.
You can get the precise reason for the deadlock by writing it's detailed
deadlock report to the SQL error log & inspecting that report. It's complex
to analyse, but if you post it back perhaps we could help you analyse it.
To write the detailed deadlock report to the error log, issue the following
command:
dbcc traceon (1204, 3605, -1)
1204 is the trace flag for detailed deadlock reports
3605 is the instruction to write that report to the sqwl error log
-1 is the instruction that the trace should apply to all connections, not
just the current connection that is issuing the dbcc traceon command.
Regards,
Greg Linwood
SQL Server MVP
"Don MacKenzie" <cd_mackenzie@.hotmail.com> wrote in message
news:2544f4a.0402131647.7bbd58cf@.posting.google.com...
> Hi,
> I have a client-server .NET system that uses an Enterprise Services
> Serviced Component (COM+ component) for data access. Under high load,
> I am getting deadlocking errors, they seem to be related to one table.
> These situations are hard to debug, but I am guessing it is because
> an update on a delete may be occurring on DIFFERENT ROWS in the same
> table at the same time. This can't be right, can it?
> I read something about problems when using indexes, but this table is
> not indexed other than the primary key. The table definition is shown
> below. Any suggestions would be appreciated.
> Thanks!
> *** TABLE DEFINITION (FIELDS HAVE BEEN RENAMED, BUT STRUCTURE IS
> ACCURATE) ***
> CREATE TABLE [Boo_Record_Foo] (
> [Boo_Id] [int] NOT NULL ,
> [Fooed_By_User_Id] [int] NULL ,
> [Fooed_By_User_Name] [varchar] (30) COLLATE
> SQL_Latin1_General_CP1_CI_AS NULL ,
> CONSTRAINT [PK_Boo_Record_Foo] PRIMARY KEY CLUSTERED
> (
> [Boo_Id]
> ) WITH FILLFACTOR = 90 ON [PRIMARY]
> ) ON [PRIMARY]
> GO
>
> *** ERROR MESSAGE ***
> Transaction (Process ID 53) was deadlocked on {lock} resources with
> another process and has been chosen as the deadlock victim. Rerun the
> transaction.

Deadlocking Limitations of SQL Server... tell me it isn't so.

Hi,
I have a client-server .NET system that uses an Enterprise Services
Serviced Component (COM+ component) for data access. Under high load,
I am getting deadlocking errors, they seem to be related to one table.
These situations are hard to debug, but I am guessing it is because
an update on a delete may be occurring on DIFFERENT ROWS in the same
table at the same time. This can't be right, can it?
I read something about problems when using indexes, but this table is
not indexed other than the primary key. The table definition is shown
below. Any suggestions would be appreciated.
Thanks!
*** TABLE DEFINITION (FIELDS HAVE BEEN RENAMED, BUT STRUCTURE IS
ACCURATE) ***
CREATE TABLE [Boo_Record_Foo] (
[Boo_Id] [int] NOT NULL ,
[Fooed_By_User_Id] [int] NULL ,
[Fooed_By_User_Name] [varchar] (30) COLLATE
SQL_Latin1_General_CP1_CI_AS NULL ,
CONSTRAINT [PK_Boo_Record_Foo] PRIMARY KEY CLUSTERED
(
[Boo_Id]
) WITH FILLFACTOR = 90 ON [PRIMARY]
) ON [PRIMARY]
GO
*** ERROR MESSAGE ***
Transaction (Process ID 53) was deadlocked on {lock} resources with
another process and has been chosen as the deadlock victim. Rerun the
transaction.COM+ tends to use the SERIALIZED isolation level which is never good for
multi-user apps. I would check to see what the isolation level is on all
the connections. You say your table has no index other than the PK
constraint. Is it ever accessed by anything other than the PK? Can you
show the 2 statements that are being used when it deadlocks?
Andrew J. Kelly
SQL Server MVP
"Don MacKenzie" <cd_mackenzie@.hotmail.com> wrote in message
news:2544f4a.0402131647.7bbd58cf@.posting.google.com...
> Hi,
> I have a client-server .NET system that uses an Enterprise Services
> Serviced Component (COM+ component) for data access. Under high load,
> I am getting deadlocking errors, they seem to be related to one table.
> These situations are hard to debug, but I am guessing it is because
> an update on a delete may be occurring on DIFFERENT ROWS in the same
> table at the same time. This can't be right, can it?
> I read something about problems when using indexes, but this table is
> not indexed other than the primary key. The table definition is shown
> below. Any suggestions would be appreciated.
> Thanks!
> *** TABLE DEFINITION (FIELDS HAVE BEEN RENAMED, BUT STRUCTURE IS
> ACCURATE) ***
> CREATE TABLE [Boo_Record_Foo] (
> [Boo_Id] [int] NOT NULL ,
> [Fooed_By_User_Id] [int] NULL ,
> [Fooed_By_User_Name] [varchar] (30) COLLATE
> SQL_Latin1_General_CP1_CI_AS NULL ,
> CONSTRAINT [PK_Boo_Record_Foo] PRIMARY KEY CLUSTERED
> (
> [Boo_Id]
> ) WITH FILLFACTOR = 90 ON [PRIMARY]
> ) ON [PRIMARY]
> GO
>
> *** ERROR MESSAGE ***
> Transaction (Process ID 53) was deadlocked on {lock} resources with
> another process and has been chosen as the deadlock victim. Rerun the
> transaction.|||Hi Don.
You can get the precise reason for the deadlock by writing it's detailed
deadlock report to the SQL error log & inspecting that report. It's complex
to analyse, but if you post it back perhaps we could help you analyse it.
To write the detailed deadlock report to the error log, issue the following
command:
dbcc traceon (1204, 3605, -1)
1204 is the trace flag for detailed deadlock reports
3605 is the instruction to write that report to the sqwl error log
-1 is the instruction that the trace should apply to all connections, not
just the current connection that is issuing the dbcc traceon command.
Regards,
Greg Linwood
SQL Server MVP
"Don MacKenzie" <cd_mackenzie@.hotmail.com> wrote in message
news:2544f4a.0402131647.7bbd58cf@.posting.google.com...
> Hi,
> I have a client-server .NET system that uses an Enterprise Services
> Serviced Component (COM+ component) for data access. Under high load,
> I am getting deadlocking errors, they seem to be related to one table.
> These situations are hard to debug, but I am guessing it is because
> an update on a delete may be occurring on DIFFERENT ROWS in the same
> table at the same time. This can't be right, can it?
> I read something about problems when using indexes, but this table is
> not indexed other than the primary key. The table definition is shown
> below. Any suggestions would be appreciated.
> Thanks!
> *** TABLE DEFINITION (FIELDS HAVE BEEN RENAMED, BUT STRUCTURE IS
> ACCURATE) ***
> CREATE TABLE [Boo_Record_Foo] (
> [Boo_Id] [int] NOT NULL ,
> [Fooed_By_User_Id] [int] NULL ,
> [Fooed_By_User_Name] [varchar] (30) COLLATE
> SQL_Latin1_General_CP1_CI_AS NULL ,
> CONSTRAINT [PK_Boo_Record_Foo] PRIMARY KEY CLUSTERED
> (
> [Boo_Id]
> ) WITH FILLFACTOR = 90 ON [PRIMARY]
> ) ON [PRIMARY]
> GO
>
> *** ERROR MESSAGE ***
> Transaction (Process ID 53) was deadlocked on {lock} resources with
> another process and has been chosen as the deadlock victim. Rerun the
> transaction.

Thursday, March 22, 2012

Deadlock problem

I've load testing a database solution and keep hitting a deadlock situation that I don't understand. I'm hoping that someone on this forum might have some solutions..

I've 3 tables: document, documentVersion and promotion table where documentVersion stores XML and the promotion table stores data extracted from the XML. I've an insertDocument stored procedure that:

    Begins a transaction Inserts a new row into document (which has an identity column as a primary key) and records the scope_identity Inserts a new row into documentVersion with a FK reference to the new document row. The documentVersion table also has an identity columan as a primary key). Again scope_identity is recorded. The name of a custom stored procedure is formed and an execute statement is used to run the stored procedure. The custom stored procedure uses XQuery to extract rows from the XML and inserts the rows into the promotion table with a FK reference to the new documentVersion. The transaction is committed.

The deadlocks always occur in step 5 and invariably are caused by process1 having an X lock on PK_documentVersion (presumably because of the insert) and waiting for a shared lock on PK_documentVersion (presumably to check the FK constraint from the promotion table insert). Process2 is in exactly the same situation (i.e. holding X lock on PK_documentVersion and waiting for shared lock on PK_documentVersion).

If I run the test without the promotion (i.e. just inserting into document and documentVersion) and build up several thousand rows then I can reenable promotion and run my load test without deadlocks.

I've tried changing lock hints on the inserts, changing the isolation level (including trying snapshot) but all to know avail.

Can anyone explain the cause of the deadlock and suggest a remedy?

Much obliged,

David.

P.S. there are clustered indexes on the identity columns of document and documentVersion.

Here is an article I wrote a few months ago regarding how to track deadlock errors with SQLDiag, a helpful tool for such a purpose. http://articles.techrepublic.com.com/5100-9592_11-6116287.html

Have you tried using the table hint READPAST in your sql statements?|||

Thanks I'll look at the article.

Unfortunately, I have no control of the shared locks because the database engine sets these because of the FK check. If I was doing a select I could use the READPAST hint. Similarly, approaches such as using READ_COMMITTED_ISOLATION or SET TRANSACTION ISOLATION LEVEL SNAPSHOT have no effect on the FK check's use of locks.

David

|||

not sure if you have resolved this now,

if not, do you have deadlock trace information? deadlocks can occur for non-obvious reasons at the auto commit level, which won't be directly apparent from the sql

Deadlock problem

I've load testing a database solution and keep hitting a deadlock situation that I don't understand. I'm hoping that someone on this forum might have some solutions..

I've 3 tables: document, documentVersion and promotion table where documentVersion stores XML and the promotion table stores data extracted from the XML. I've an insertDocument stored procedure that:

    Begins a transaction Inserts a new row into document (which has an identity column as a primary key) and records the scope_identity Inserts a new row into documentVersion with a FK reference to the new document row. The documentVersion table also has an identity columan as a primary key). Again scope_identity is recorded. The name of a custom stored procedure is formed and an execute statement is used to run the stored procedure. The custom stored procedure uses XQuery to extract rows from the XML and inserts the rows into the promotion table with a FK reference to the new documentVersion. The transaction is committed.

The deadlocks always occur in step 5 and invariably are caused by process1 having an X lock on PK_documentVersion (presumably because of the insert) and waiting for a shared lock on PK_documentVersion (presumably to check the FK constraint from the promotion table insert). Process2 is in exactly the same situation (i.e. holding X lock on PK_documentVersion and waiting for shared lock on PK_documentVersion).

If I run the test without the promotion (i.e. just inserting into document and documentVersion) and build up several thousand rows then I can reenable promotion and run my load test without deadlocks.

I've tried changing lock hints on the inserts, changing the isolation level (including trying snapshot) but all to know avail.

Can anyone explain the cause of the deadlock and suggest a remedy?

Much obliged,

David.

P.S. there are clustered indexes on the identity columns of document and documentVersion.

Here is an article I wrote a few months ago regarding how to track deadlock errors with SQLDiag, a helpful tool for such a purpose. http://articles.techrepublic.com.com/5100-9592_11-6116287.html

Have you tried using the table hint READPAST in your sql statements?|||

Thanks I'll look at the article.

Unfortunately, I have no control of the shared locks because the database engine sets these because of the FK check. If I was doing a select I could use the READPAST hint. Similarly, approaches such as using READ_COMMITTED_ISOLATION or SET TRANSACTION ISOLATION LEVEL SNAPSHOT have no effect on the FK check's use of locks.

David

|||

not sure if you have resolved this now,

if not, do you have deadlock trace information? deadlocks can occur for non-obvious reasons at the auto commit level, which won't be directly apparent from the sql